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Reporting System (NRS) educational functioning levels and support valid and 
reliable measures of educational gain by using standardized performance 
assessments. During year one the following were accomplished: (1) more than 

100 teachers in 5 states were trained in developing instruction and assessing 
performance on the EFF standards; (2) a set of instructional and 
documentation tools that help teachers embed assessment in on-going 
instruction were developed; (3) more than 300 teacher-generated performance 
tasks for 10 EFF standards were collected; and (4) performance tasks and 
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create draft performance continue for 5 standards. The EFF/NRS project has 
promoted professional development by enabling practitioners to build 
knowledge and skills in teaching with standards; create and use learning 
activities with embedded assessment; and observe, document, and interpret 
learner performance. The project has led to a strong foundation for adult 

improvement through strengthened capacity for 
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Executive Summary 



T he Equipped for the Future/National 
Reporting System (EFF/NRS) Data 
Collection Project is a multiyear 
national project jointly funded by the 
National Institute for Literacy (NIFL) 
and the US. Department of Education (USED). It 
was developed to create strong linkages between the 
Equipped for the Future (EFF) Standards and the 
National Reporting System (NRS), which was estab- 
lished by the USED’s Office of Vocational and Adult 
Education (OVAE) to meet accountability provi- 
sions for Title II of the Workforce Investment Act 
(WIA). 

The goal of the project is to enable programs 
using Equipped for the Future as a framework for 
instruction to report student progress from level to 
level on the National Reporting System. Specific 
outcomes for the EFF/NRS Data Collection Project 
include being able to 

• identify EFF Performance Tasks that represent the 
knowledge and skills necessary for transition from 
one NRS level to the next for up to 10 EFF Stan- 
dards, 

• make sure that transition tasks identified for each 
EFF Standard are appropriate for ABE (adult and 
basic education) and ESL (English as a second lan- 
guage) learners, and 

• develop a rich body of performance descriptors 
for each NRS level for each EFF Standard. These 



can be used to validate and enrich the existing 
body of level descriptors so that they are robust 
enough to support a standardized approach to 
assessment and reporting. 

The project is being conducted in partnership 
with five states: Maine, Ohio, Oregon, Tennessee, 
and Washington. These states have supported the 
participation of local programs as research sites in 
exchange for technical assistance from EFF, includ- 
ing training in the use of products, tools, and proce- 
dures that support valid and reliable measures of 
educational gain using standardized performance 
assessments. 

Now in its second year, the project is the 
National Institute for Literacy’s primary vehicle for 
assuring that the EFF Assessment Framework is 
aligned with actual student performance as well as 
with cognitive science research on cognition, learn- 
ing, and the development of expertise. 

The EFF Assessment Framework 

The Equipped for the Future Assessment Frame- 
work defines levels of performance and measures of 
performance on the EFF Content Standards for a 
variety of assessment purposes. The Framework 
describes adult performance along four dimensions: 
• increasing knowledge, organization, and ability to 
apply knowledge and strategies (structure of the 
knowledge-base dimension). 
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• increasing fluency in performance (fluency of per- 
formance dimension), 

• increasing independence in performance (inde- 
pendence of performance dimension), and 

• ability to perform tasks of increasing complexity 
under a variety of conditions (range of conditions 
for performance dimension). 

When completed, the EFF Assessment Frame- 
work will include 

• a developmental sequence of descriptions of 
learner performance for each of the 16 EFF Stan- 
dards that can be used to guide learning and 
instruction; 

• model performance-based assessments and scor- 
ing guidelines (rubrics) for each EFF Standard that 
can be used to mark transitions from one level of 
performance to the next (for at least 6 specified 
levels to correspond to the current 6 ABE/ASE 
(adult secondary education) and 6 ESL NRS Edu- 
cational Functioning Levels); and 

• materials, training, and technical assistance to 
support the implementation of these EFF-based 
curriculum and instructional resources and assess- 
ment tools. 

These aspects of the EFF Assessment Frame- 
work will enable us to enrich the NRS Educational 
Functioning Levels and support valid and reliable 
measures of educational gain by using standardized 
performance assessments. 

This report focuses on achievement of Year One of 
the joint Data Collection Project, October 2000- 
October 2001. 

Results 

During Year One, the EFF/NRS Data Collection 
Project 

• trained more than 100 teachers in 5 states in devel- 
oping instruction and assessing performance on 



the EFF Standards, 

• developed a set of instructional and documenta- 
tion tools that help teachers embed assessment in 
on-going instruction, 

• collected more than 300 teacher- generated Perfor- 
mance Tasks for 10 EFF Standards, and 

• analyzed Performance Tasks and descriptions of 
adult learner performances on these tasks to create 
draft performance continua for 5 Standards. 

Through these field research activities, the 
project constructed draft continua of performance 
for five EFF Standards: Read With Understanding, 
Convey Ideas in Writing, Speak so Others Can Under- 
stand, Listen Actively, and Use Math to Communicate 
and Solve Problems, These research activities also 
enabled the project to amass preliminary perfor- 
mance data for five additional Standards: Solve 
Problems and Make Decisions, Cooperate With Oth- 
ers, Learn Through Research, Take Responsibility for 
Learning, and Use Information and Communications 
Technology, 

Improving Instruction, Accountability, 
and Continuous Improvement 

The EFF/NRS project serves as an excellent profes- 
sional development experience for participating 
practitioners. It enables them to build knowledge 
and skills in 

• teaching with standards; 

• creating and using learning activities with embed- 
ded assessment; and 

• observing, documenting, and interpreting learner 
performance. 

The Data Collection Project has also created a 
strong foundation for adult education system 
reform and improvement. The combination of pro- 
fessional development for a cadre of adult educators 
(who can, in turn, train others) and the develop- 
ment of tools (Performance Tasks and performance 
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continua) that facilitate use of assessment data in 
monitoring and improving instructional practices 
has strengthened the capacity of adult education 
systems for accountability and continuous improve- 
ment. 

The promise of accountability leading to edu- 
cational improvement can be realized when teachers 

• recognize the value of monitoring adult student 
learning, 

• have appropriate and practical tools to monitor 
learning outcomes, 

• are able to make use of learner performance data 
to improve instruction, and 

• can accurately assess student progress and report 
learning gains. When these goals are met, the 
promise of accountability leading to educational 
improvement can be realized. 

Remaining Chaiienges 

Work on the development of the EFF Assessment 
Framework has been a complex and challenging 



endeavor. Our original timeline for developing per- 
formance continua for the 16 EFF Standards has 
had to be revised as we identified more clearly the 
research, analysis, and validation needed to produce 
quality results. 

At the same time, we have become more aware 
of the value of the practical tools and professional 
development provided in the course of our develop- 
ment work. Teacher/researchers who have partici- 
pated in the project have a better understanding of 
evidence and of the conditions for assessment — the 
big picture of assessment. They constitute a core of 
teachers who are better equipped to provide reliable 
data on learning outcomes. 

These interim results are critical to meeting 
the broadest goals of the EFF initiative as well as the 
NRS, which is continuous improvement of the adult 
education and literacy system. With continued com- 
mitment from our field partners and increased 
interest from other states, we are looking forward to 
seeing this work through to completion. 




3 



8 



EFF/NRS DATA COLLECTION PROJECT. 2000-2001 



Guiding Principles for Developing 
EFF Assessment Framework 



1. The EFF Assessment 
Framework must address multi- 
ple purposes for assessment. 
The Framework must provide for 

• information on learner achieve- 
ments and mastery that is 
useful to the learner as well as 
the teacher throughout the 
instructional process; 

• information about what learners 
can do that is credible to 
employers, educational 
institutions, and policymakers, 
as well as to learners them- 
selves; and 

• information that is useful for 
program and system improve- 
ment and accountability. 

2. To address these multiple 
purposes, the EFF Assessment 
Framework must support a 
multidimensional, flexible, and 
systemic approach to assess- 
ment. Teachers and programs will 
be able to choose from a range 
of tools, to be identified or 
developed, that enable them to 
accurately measure performance 
against EFF Standards and that 
are linked to one another so 

that multiple assessments can 
provide a rich portrait of learner 
competence. 

3. The EFF Assessment Frame- 
work must address learning 
over a lifetime. Strategies for 
assessment and credentialing 
must take into account the fact 
that adults build skills over time 
(rather than all at once) in 



response to changes in their life 
situations. Certificates and other 
credentials must be modular, 
designed to define competence or 
mastery at a particular point, and 
within a framework that assumes 
continuing development of 
competence as skills, knowledge, 
and understanding are further 
developed over time. 

4. Since EFF Standards define 
skills all adults need in order to 
carry out their roies as workers 
and as members of families and 
communities, the EFF Assess- 
ment Framework must address 
a single continuum of perfor- 
mance for ali adults — adults with 
only minimal formal education and 
those with many years of formal 
education, including advanced 
degrees. 

5. Each level defined in the EFF 
Assessment Framework must 
communicate clearly what an 
adult at that level can do. 

Numerical levels don’t communi- 
cate meaning to external 
audiences. Grade levels seem to 
communicate a common picture 
of performance; but, in fact, the 
meaning behind the label varies 
widely from community to commu- 
nity and state to state. Grade levels 
are particularly misleading when 
applied to adult performance, 
since they focus on developmental 
skill levels that don’t match the 
ways in which adults, with their 
broader background and range of 



experience, can combine skills and 
knowledge to perform effectively in 
daily life. 

6. The levels defined in the 
EFF Assessment Framework 
must be explicitly linked to key 
external measures of compe- 
tence (e.g., certificates of 
mastery, NAAL7IAL survey levels, 
diplomas, and other credentials) 
and key pathways (e.g., entry to 
higher education and entry to 
employment as defined by 
occupational skill standards) so 
that adults and systems can rely 
on them as accurate predictors 
of real-world performance. 

7. The levels defined in the EFF 
Assessment Framework must 
be the product of a national 
consensus-building process 
that assures portability of 
certificates and credentials. 

8. Work on the development of 
this framework must maintain 
the strong customer focus 
that has distinguished the EFF 
Standards development process 
to date. It must be based on a 
broad, inclusive definition of 
maximizing accountability for all 
activities to all customers, starting 
with the adult learner. 
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Introduction 



I n the summer of 1999, the National Institute 
for Literacy (NIFL) and the Division of 
Adult Education and Literacy (DAEL) of the 
U.S. Department of Education*s (USED) 
Office of Vocational and Adult Education 
(OVAE) began a series of meetings to discuss how to 
align work on Equipped for the Future (EFF) Stan- 
dards with work on the National Reporting System 
(NRS) being developed to meet reporting require- 
ments on the Adult Education and Family Literacy 
Act (AEFLA), Title II of the Workforce Investment 
Act (WIA). By the following summer, NIFL and 
DAEL had developed a plan for a joint project 
intended to create strong linkages between the EFF 
Standards and the National Reporting System. The 
goal of the project was to enable programs using 
Equipped for the Future as a framework for instruc- 
tion to report student progress from level to level on 
the NRS. The project would work with states that 
had chosen not to use an existing standardized 
instrument to report progress on the NRS by pro- 
viding technical assistance that included training in 
.the use of products, tools, and procedures that 
would support valid and reliable measures of educa- 
tional gain using standardized alternative assess- 
ments. Specific outcomes proposed for the EFF/ 
NRS Data Collection Project were to 
• identify EFF Performance Tasks that represent the 
knowledge and skills necessary for transition from 



one NRS level to the next for up to 10 EFF Stan- 
dards, 

• identify transition tasks for movement between all 
6 adult basic education (ABE) levels and all 6 Eng- 
lish as a second language (ESL) levels on the 
National Reporting System for each EFF Standard, 
and 

• develop a rich body of performance descriptors 
for each NRS level for each EFF Standard. These 
would be used to validate and enrich the existing 
body of level descriptors so that they are robust 
enough to support a standardized approach to 
assessment and reporting. 

This final outcome was of particular impor- 
tance in assuring a standardized approach to assess- 
ment within and across states. The performance 
level descriptors for the NRS Educational Function- 
ing Levels for ABE and ESL had been designed for 
illustrative purposes only and are not specific 
enough to provide a basis for developing alternative 
assessment tasks and scoring guidelines. However, 
states that had chosen to give adult education pro- 
grams an alternative to assessing educational gains 
using existing standardized test instruments were 
relying upon these instruments to determine 
whether students possessed the skills and knowledge 
necessary to move from one level to another. DAEL 
entered into partnership with NIFL on the EFF/NRS 
Data Collection Project because they saw that our 
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work in collecting data to build performance con- 
tinua for each EFF standard could result in the 
research-based construct and research-based level 
descriptors that would support valid and reliable 
performance assessment. 

The joint EFF/NRS Data Collection Project 
was launched in October 2000 with funding from 
DAEL, NIFL, and the five state adult education 
agencies that have been partners in this data collec- 
tion effort: Maine, Ohio, Oregon, Tennessee, and 
Washington. In October 2001, a second year of the 
Data Collection Project was funded. This project is 
the primary vehicle for assuring that the continuum 
of increasingly skilled performance (described on 
page 8) constructed for each EFF Standard is 
aligned with actual student performance as well as 



with cognitive science research on cognition, learn- 
ing, and the development of expertise.^ The project 
also serves as an excellent professional development 
vehicle for participating practitioners, enabling 
them to build knowledge and skills in teaching with 
standards; creating and using learning activities 
with embedded assessment; and observing, docu- 
menting, and interpreting learner performance. 

This report provides an overview of the 
accomplishments of the first year of the EFF/NRS 
Data Collection Project.^ It includes a discussion of 

• work carried out from October 2000-September 
2001, 

• the field research process and the practice-based 
learning of participants, and 

• key lessons and their implications for the NRS. 



‘See John D. Bransford, Ann L. Brown, and Rodney Cookings, Eds. (1999). How People Learn: Brain, 
Mind Experience and School. (Washington DC: National Academy Press); Nadine M. Lambert and 
Barbara L. McCombs, Eds. (1998). How Students Learn: Reforming Schools Through Learner-Centered 
Education (Washington DC: America Psychological Association); and Jennifer Cromley, Learning to 
Think, Learning to Learn: What the Science of Thinking and Learning has to Offer Adult Education (2000) 
(Washington, DC: National Institute for Literacy). 

^For the first part of the project year, the work was managed by the Center for Literacy Studies, Univer- 
sity of Tennessee, with technical assistance provided by SRI International. In April 2001, a new EFF 
Assessment Consortium was awarded a contract from NIFL to complete the work of building the EFF 
Assessment Framework. The Consortium is a partnership between SRI and CLS, guided by a Technical 
Advisory Group, fully listed in Appendix A. 
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Background on the Development of 
the EFF Assessment Framework, 

1999-2000 



I n January 1999, as NIFL/EFF was conduct- 
ing the third and final field review of the 
EFF Content Standards, we took the first 
steps toward developing an assessment 
framework for the Standards. We began by 
reviewing analogous efforts in England, Australia, 
and South Africa. We also commissioned a series of 
papers focused on issues important to assuring that 
programs and states could reliably and validly assess 
and report progress in relation to the EFF Stan- 
dards. Intended primarily for internal use by the 
EFF team, these papers included 
• a broad look at the issues of developing national 
assessments and national performance standards 
by Archie LaPointe of the Educational Testing 
Service, 

• a “road map ’ for a process for developing perfor- 
mance standards for EFF by Regie Stites of SRI 
International, 

• a review by Michelle Della Rosa of HumRRO and 
Joan Wills of the Institute for Educational Leader- 
ship (lEL) that looked more closely at the con- 
struct of each of the EFF Standards, and 
• a paper by Sri Ananda of WestED intended to 
assist teachers in assessing student performance 
for in-program purposes using the EFF Standards. 
This paper. How Instructors Can Support Adult 



Learners Through Performance-Based Assessment is 
the only one prepared for general distribution. 

At the same time, we invited our field develop- 
ment partners to work with us to clarify the various 
purposes a comprehensive assessment system for 
adult education needs to address^ and to help us 
develop a set of Guiding Principles for Developing 
the EFF Assessment Framework (see page 4). Taken 
together, our Guiding Principles all pointed to using 
cognitive science research on the development of 
expertise as the theoretical underpinning for the 
EFF Assessment Framework. This research base 
enabled us to conceptualize a single continuum of 
increasingly skilled performance that includes all 
adult performance — from novice to expert. It also 
provided a starting point for defining a small num- 
ber of key dimensions that distinguish perfor- 
mances along the continuum. 

Defining the EFF Continuum of Performance 
was the first of three primary tasks toward fully 
developing the EFF Assessment Framework. The 
remaining tasks were 

• to develop a continuum of performance for each 
standard, with levels that describe real-world com- 
petence and 

• to identify and develop tools to assess performance 
for the range of assessment purposes. 



^See Appendix B for our Purposes of Assessment chart, adapted from RJ. Mislevy (1994). 
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Defining the EFF Continuum 
of Performance 

In order to assure that our model of performance 
reflected real teaching and learning situations, in the 
spring of 1999, we invited teachers at 10 EFF field 
development sites, who were already using the Stan- 
dards to plan and guide instruction, to also docu- 
ment and evaluate learner performance."* Teacher 
reports on learner progress were analyzed by using 
the research on expertise to clarify which Dimen- 
sions of Performance were important to consider in 
rating how well an individual could use an EFF skill 
to carry out real-life tasks. 

Through this review process, we identified 
four key dimensions that characterized progress in 
using a Standard along a continuum.^ These 
Dimensions of Performance, validated through data 
collected by EFF field sites in 1999 and 2000, includ- 
ed the following: 

1 . Structure of Knowledge Base, 

2. Fluency of Performance, 

3. Independence of Performance, and 

4. Range of Conditions for Performance. 

Our initial conception of this continuum was 
represented by a graphic that showed independence. 



fluency, and range of performance increasing as the 
knowledge base deepened (see Figure 1). 

Developing a Continuum of Performance 
for Each EFF Standard 

By the end of 1999, we were ready to begin the 
process of building performance continua based on 
these four dimensions. Teachers and tutors from 15 
field sites in our five partner states used standard- 
ized performance templates and a standard data col- 
lection reporting protocol to collect and report data 
on student performance. The protocol guided 
teachers through a process of using the standards 
and dimensions to plan and carry out lessons cen- 
tered on EFF “learning tasks” and to collect evidence 
of learner performance in relation to these tasks. 
Each teacher was responsible for reporting on only 2 
of the 16 Standards — one Standard from the com- 
munication group and one from another group. 
EFF staff worked with all the sites to make sure that 
at least three teachers were collecting data on each 
standard — more in the case of the communication 
standards. 

Review of this first round of data, followed by 
a midcourse technical assistance meeting with prac- 



^See Sondra G. Stein (2000) Equipped for the Future Content Standards: What Adults Need to Know and 
Be Able to Do in the 21st Century. (Washington, DC: National Institute for Literacy), See Appendix B, for 
a full description of the documentation protocol. 

^Primary sources were Wittrock and Baker (1988) Testing and Cognition (Englewood Cliffs: Prentice 
Hall), Bransford et al., How People Learn: Brain, Mind Experience and School (Washington DC: National 
Academy Press), and a wide range of technical reports from the National Center for Research on 
Evaluation, Standards, and Student Testing (CRESST) at UCLA. 

®For more information on the EFF Dimensions see Equipped for the Future Content Standards 
pp. 59-60 and Peggy M McGuire (2000) “A Performance Framework for Teaching and Learning With 
the Equipped for the Future (EFF) Content Standards,” Adventures in Assessment, Vol.l2, Winter, 2000, 
pp. 28-43 (Boston: SABES/World Education). 
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Figure 1. EFF Performance Continuum 
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titioners in the spring of 2000, enabled us to 
improve the technical quality of our data collection 
methods in three areas crucial for the work of the 
current EFF/NRS Data Collection Project: 

1. Sharpening the focus on the “construct” or con- 
ceptual model of each EFF Standard. Data from 
teachers helped us understand that a well-struc- 
tured Performance Task must focus equally on 
the full Standard and on the Dimensions of Per- 
formance. In their effort to create tasks that pro- 
vided opportunities for students to develop along 
all four dimensions of performance, teachers at 
first lost focus on the Standard that was the target 
of their effort. They created interesting, real- 
world performance tasks, which often did not, in 
fact, provide evidence of use of the Standard on 
which they intended to focus. We clarified that 



the construct for each Standard is composed of 
the Components of Performance of the Standard 
(i.e„ the full Standard statement) as modified by 
the four Dimensions of Performance. 

2. Sharpening the distinction between the require- 
ments of a task created to elicit evidence of per- 
formance and the descriptions of learner 
performance on this task. To help clarify this 
critical distinction, we worked with our field 
partners to develop criteria for both defining a 
welUstructured Performance Task and generating 
rich descriptions of learner performance. Place- 
ment of a performance-task description on the 
continuum does not vary from learner to learner 
and is determined by the complexity and context 
of the task, including the knowledge required to 
perform it. The placement of descriptions of 
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learner performance on the continuum does vary 
from learner to learner and is determined by the 
knowledge, strategies, fluency, and independence 
demonstrated in performance on a task. 

3. Increasing interrater reliability on the placement 
of tasks and learner performance on the contin- 
uum. Working through the issues described above 
gave teachers a set of more objective criteria for 
deciding where to place tasks and descriptions of 
learner performance on a continuum. In the mid- 
course technical assistance meeting, we practiced 
coming to consensus on placing tasks on the con- 
tinuum and developed increased ability to judge 
both tasks and performance in light of explicit, 
common criteria. 

Through this collaborative work with field 
sites, we refined data collection procedures to assure 
that teachers and tutors paid close attention to 
whether the learning tasks they were developing 
were well structured and that descriptions of learner 
performance on an EFF Performance Task focused 
clearly on how well a student can use the targeted 
skill (as defined by the Standard) in carrying out a 
task. To assure that we were collecting data based on 
the same construct of a specific Standard, we created 
templates that provided teachers with a common 
language for describing tasks and learner perfor- 
mance relative to the four dimensions. These refine- 



ments in our data collection practices enabled us to 
be certain that the data we were collecting could be 
aggregated to build a valid continuum of perfor- 
mance for that Standard. 

By the summer of 2000, the EFF Assessment 
Team had built a strong foundation for the 
EFF/NRS Data Collection Project. We had defined 
the EFF Purposes of Assessment (Appendix B), 
identified the four Dimensions of Performance 
upon which to build a cognitive model of adult per- 
formance, and begun the process of observing stu- 
dent performance of EFF Standards by using a 
performance framework and template based on 
these four dimensions. We had revised our data col- 
lection protocols based on feedback from our part- 
ner teachers/researchers, and we had prepared a 
guide for EFF field researchers that focused much 
more clearly on the importance of basing EFF per- 
formance data on well-structured Performance Tasks. 
The rich body of data and data collection tools we 
had developed prepared us to create a partnership 
with the Division of Adult Education and Literacy 
of the U.S. Department of Education’s Office of 
Vocational and Adult Education and to continue the 
process of building the continua with the clear 
intention of aligning the EFF assessment system 
with the U.S. Department of Education’s National 
Reporting System. 
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Overview of the EFF/NRS 
Data Collection Project, Year One 



T he scope of work for the first year of 
the EFF/NRS Data Collection Project 
was ambitious: Use student perfor- 
mance data collected by practitioners 
to build a continuum of performance 
for up to 10 of the 16 EFF Standards in order to pro- 
vide research-based performance descriptors to 
enrich NRS level descriptors. We planned to build 
each continuum by arraying descriptions of real-life 
applications of the knowledge and skills embedded 
in a standard from a beginning level toward devel- 
oping expertise. We would then use this perfor- 
mance data to create behavioral descriptors that 
would be aligned with levels on the NRS. In this way 
we could “anchor” these key NRS transition points 
(or levels) along an EFF Standard continuum. The 
10 Standards that were jointly identified as impor- 
tant to focus on first were: Read With Understand- 
ing, Convey Ideas in Writing, Speak so Others Can 
Understand, Listen Actively, Use Math to Solve Prob- 
lems and Communicate, Solve Problems and Make 
Decisions, Cooperate With Others, Take Responsibility 
for Learning, Learn Through Research, and Use Infor- 
mation and Communications Technology. 

This work was carried out in collaboration 
with the same five state agency partners engaged in 
earlier data collection work. All expressed a strong 



interest in working with us to integrate EFF Stan- 
dards into assessment procedures that can be used 
for reporting to the U.S. Department of Education. 
Twenty- four field sites were identified by the state 
adult education offices in Ohio, Oregon, Maine, 
Tennessee, and Washington according to criteria 
developed by NIFL/DAEL. These field sites repre- 
sent a broad spectrum of adult education students 
and providers. They include Even Start, family liter- 
acy, and workplace literacy programs in a wide 
range of learning sites, including school-based cen- 
ters, community-based organizations, and commu- 
nity colleges. They also include adult basic 
education (ABE) and English as a second language 
(ESL) students who participate in instruction at all 
six ABE and all six ESL levels of the NRS. The state 
agency partners supported (both financially and 
programmatically) the participation of more than 
80 teachers in these programs. Funding from NIFL 
and DAEL covered all expenses associated with 
training and technical assistance, which included 
two national meetings and meetings in each state. In 
order to build in-state capacity on assessment issues, 
we decided to use these resources to train and sup- 
port a part-time field assistant in each state rather 
than expand our own staff Over time, the field 
assistants assumed increasing responsibility for pro- 




1 1 



EFF/NRS DATA COLLECTION PROJECT. 2000-2001 



viding on-going technical assistance to field sites. 
The timetable for Year One included 

• initial training, 

• three data reporting periods, 

• a midcourse meeting, 

• a data analysis period from June to August to con- 
struct the draft standards — specific performance 
continua, and 

• a final national meeting in September for the 
research teams to review and validate the draft 
continua. 

Teacher and Program 
Administrator Agreements 

Each participating teacher agreed to prepare and 
submit a total of six data reports, using a common 
protocol and reporting form on two Standards. 
Each report described in detail an EFF Performance 
Task; four of the reports also documented the per- 
formance of at least three students on the task. Pro- 
gram administrators agreed to convene regular 
meetings of their program research team for the pur- 
pose of discussing task “ratings” and descriptions of 
student performance and to provide support for 
teachers who were often trying out a very different 
approach to teaching than they had been accus- 
tomed to using. Program administrators reported 
quarterly on their observations of changes in teach- 
ing patterns, student learning, and overall impact of 
EFF field research on their program. (See Appendix 
A for the detailed timeline of project activities and a 
sample agreement form that was signed by all partic- 
ipants.) 

October-December 2000: 

The First Reporting Period 

For the first 2 months, teachers worked on writing 
and analyzing performance tasks that met the crite- 



ria for a well-structured task. These tasks were 
reviewed by the field assistants and revised by teach- 
ers. Many teachers used these performance tasks 
with students, but we did not ask them to document 
student performance at this time. After having iden- 
tified the importance of well-structured tasks to 
building a valid continuum of performance for each 
Standard, we kept the focus squarely on building 
teacher capacity to develop well-structured perfor- 
mance tasks. 

January-June 2001 

During the two subsequent reporting periods (end- 
ing in April and June), teachers developed addition- 
al tasks and, using the performance template as a 
guide, placed descriptions of learner performance at 
points along the continuum of developing expertise. 
These descriptions were submitted electronically to 
the Center for Literacy Studies, the Consortium 
partner that manages the data archives. (See Appen- 
dix C for a copy of the reporting form). 

For example, to collect evidence of student per- 
formance of the Standard Read With Understandings 
teachers created performance tasks that incorporated 
all of the components of this standard and repre- 
sented one instance of meaningful, real-world use of 
Read With Understanding for students in their class- 
es. They analyzed these tasks along the four dimen- 
sions and placed the task descriptions at the 
appropriate point along the continuum of increasing 
complexity, using the tools described in Section 3, 
Training, Technical Assistance, and Practice-Based 
Learning in the Field Development Process. Student 
performance of these tasks was then described in 
detail and placed along the continuum, defined by 
the four dimension, of developing a knowledge base 
and increasing fluency, range, and independence. An 
example of a performance task and a description of 
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COMMUNICATION SKILL 

Read With Understanding 

In order to fulfill responsibilities as parents/family members, 
citizens/community members, and workers, adults must be able to: 



STANDARD 



Read With 
Understanding 




Student performance following the reporting proto- 
col are found in Appendix C* 

July^^ptember 2001 

By the end of the data collection period, the project 
staff had received between 12 and 34 data reports 
for each of the ten standards for which we were col- 
lecting performance data. Staff and field assistants 
reviewed and analyzed all data and worked in small 
groups to begin to create draft standard-specific 
performance templates for the standards on which 
the NRS focuses: Read With Understanding, Convey 
Ideas in Writing, Listen Actively, Speak So Others Can 



Understand, and Use Math to Solve Problems and 
Communicate. 

Four of these continua and the data templates 
from which they were developed were analyzed at 
an end-of-project national meeting of field develop- 
ment partners (including state agency representa- 
tives, program administrators, and teachers) in 
September 200 L Due to the events of September 1 1, 
not all project participants were able to participate 
in this review, so the review period was extended. At 
a national meeting held in early November to initi- 
ate the next phase of data collection, field develop- 
ment partners had the opportunity to review 




13 




EFF/NRS DATA COLLECTION PROJECT, 2000-2001 



versions of the continua that incorporated recom- 
mendations from the September meeting. 

October 2001 -September 2002: 

Year-lWo Plans 

These standard-specific performance continua are 
being prepared for technical review by expert panels 
in early 2002. During the 3-day review process, pan- 
elists with a variety of backgrounds and expertise (in 
teaching, subject matter, assessment, and policy) will 
engage in a systematic review of the performance 
data and level descriptions, using guidelines adapted 
from accepted procedures developed for establishing 
performance levels for performance assessments. 
Panelists will make suggestions for revisions to the 
levels of the performance continuum and for revi- 
sions to the assessment task specifications. 

By September 2002, we will have EFF perfor- 
mance-level descriptors for five Standards and per- 
formance-level indicators for the NRS Educational 
Functioning Levels. In the fall of 2002, we will con- 



vene an assessment task development institute that 
will make use of the revised performance continua 
for these five Standards and develop model assess- 
ment tasks that can be used for assessing perfor- 
mance on the Standards for accountability 
purposes. During the winter and spring of 
2002-2003, we intend to conduct pilot studies of the 
use of model assessment tasks for reporting educa- 
tional gains on the NRS. During this same time 
period, 2002-2003, we will be reviewing and revis- 
ing the performance continua for the next group of 
standards. By the spring of 2004, we intend to have a 
range of performance assessment products available 
for policymakers and practitioners. This projected 
timetable through 2004 is included in Appendix A. 

In the following section, we describe the data 
collection process for this first year of the EFF/NRS 
Data Collection Project (October 2000-September 
2001) in more detail through an explanation of the 
tools that were used to collect the data and the 
processes in which teachers engaged. 
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Training, Technical Assistance, 
and Practice-Based Learning in the 
Field Development Process 



F rom the beginning, the aim of the 
EFF/NRS Data Collection Project was 
to develop products, tools, and proce- 
dures that would enrich the NRS Edu- 
cational Functioning Level descriptions 
and that would support valid and reliable measures 
of educational gain using standardized alternative 
assessments. For states that have chosen to give 
adult education programs an alternative to assess- 
ing educational gains with existing standardized 
test instruments, the currendy available qualitative 
descriptions of levels of knowledge, skills, and abili- 
ties in the NRS Educational Functioning Levels are 
not specific enough to provide a basis for develop- 
ing alternative assessment tasks and scoring guide- 
lines. Recognizing that the current descriptors were 
not designed to be used for assessment purposes, 
USED/DAEL and NIFL jointly sponsored the 
EFF/NRS Data Collection Project as a means of col- 
lecting and analyzing adult learner performance 
data that could be used to create more specific and 
detailed qualitative descriptors for each NRS skill 
area. The expected outcomes of the joint project 
were enriched performance-level descriptors and a 
set of performance assessment tasks that could be 
used to mark the transition from level to level on 
the NRS. 

Beyond these specific expected outcomes, there 
were several additional benefits expected from the 
EFF/NRS Data Collection Project. Among these 



expected benefits are the possibilities of developing 
tools for measuring educational gains that are more 
sensitive to smaller increments of change and more 
closely aligned with adult learner and adult educa- 
tion program goals than those found in currently 
available standardized tests. Also, as adult education 
moved to the more rigorous accountability require- 
ments of the AEFLA (Tide II, WIA), there was wide 
concern that insufficient training in assessment pro- 
cedures used by practitioners in the field would lead 
to problems in the validity and reliability of the edu- 
cational gains data required for accountability. 
DAEL and NIFL believed that the considerable train- 
ing and technical assistance that was part of the EFF 
field-based research procedures would result in the 
states in which we are working in a cadre of practi- 
tioners more experienced in assessment. Moreover, it 
was believed that the EFF field-based research would 
produce a set of tools that would assist other practi- 
tioners and other states in putting into place a more 
rigorous approach to assessment and reporting. 

Teacher/researchers in the EFF/NRS Data Col- 
lection Project received training and tools to help 
develop their expertise in the specific tasks required 
for the research process by 

• constructing a performance task that targeted one 
of the EFF Standards and that was appropriately 
challenging for the students’ skill levels, 

• developing a clear and detailed picture (under- 
standing) of the knowledge base required to per- 
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form the task so that the task could be used as a 
guide for instruction as well as an opportunity to 
collect evidence of performance, 

• constructing tools to collect evidence of student 
performance on the task in relation to the four 
Dimensions of Performance, and 

• developing descriptions of student performance, 
relative to the four dimensions, that were suffi- 
ciendy detailed to enable multiple practitioners to 
reach agreement on where to place the perfor- 
mance on a continuum of such performances. 

In order to assist practitioners in collecting 
and reporting the kind of rich and useful data that 
we would need to gather in the proposed research 
process and in response to what we learned about 
that process from the practitioners along the way, 
we developed several tools. Below, we describe these 
tools in more detail, explaining why we developed 
them and how teachers used them in the data col- 
lection process. A copy of each form or tool is found 
in Appendix B. The tools include 

• a performance template, which is an array of 
increasingly complex generic descriptors of per- 
formance along all four dimensions; 

• a definition of an EFF Performance Task and a 
graphic that embeds the Performance Task in the 
teaching and learning process; 

• a task template consisting of an array of increas- 
ingly complex generic descriptors for the knowl- 
edge that a task requires and for the conditions of 
performance of the task to assist teachers in rating 
the objective complexity of the task; 

• a Performance Task Worksheet to guide teachers 
through the process of creating and analyzing a 
performance task; 

• a set of criteria for well-constructed performance 
tasks; and 

• teacher and student observation forms. 



The Performance Template: 

A Key Tool for Building the Performance 
Continuum 

The Guiding Principles for the EFF Assessment 
Framework called for a single continuum of perfor- 
mance for all adults. In envisioning such a continu- 
um, we drew on our knowledge of research on adult 
cognition and learning. We wanted to build a “con- 
tinuum” for each EFF standard that represented a 
rich developmental picture of adult applied knowl- 
edge and skills — a picture that included all adults, 
that encompassed a lifetime of learning, and that 
articulated learning as movement from novice to 
expert performance. Our work on the NRS/EFF 
Data Collection Project focuses only on that part of 
each continuum that represents the performance of 
adults participating in the current adult literacy/ 
basic education/ESL system and aims to ensure that 
each continuum reflects what adult performance 
really looks like in the classrooms and instructional 
settings of that system. 

The EFF Performance Template was the start- 
ing point for using field data to construct a develop- 
mental performance continuum based on the four 
dimensions. It provided teachers — researchers with 
a set of research-based generic criteria to guide and 
standardize the process of placing detailed observa- 
tions of ABE and ESL learner performance at appro- 
priate points along a developmental performance 
continuum from novice to expert. The Performance 
Template is organized around three observation cat- 
egories — Knowledge Base, consisting of what learn- 
ers know; Performance, including both fluency and 
independence; and Range, describing the kind and 
number of tasks and contexts — that reflect the four 
dimensions (see Figure 2). 

Performance descriptions keyed to these obser- 
vation categories were presented in 10 consecutive 
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Figure 2. EFF Performance Template Questions 



GUIDING PRINCIPLES FOR DEVELOPING THE EFF ASSESSMENT FRAMEWORK 



DIMENSIONS OF PERFORMANCE 



Knowledge Base 

What do learners 
know? 


1 . What vocabulary do learners have related to the skill? Related to the subject area? 

2. What content knowledge do learners have related to the skill? Related to the subject area? 

3. What strategies do learners have for organizing and applying content knowledge? 

• Can learners recognize or create new relationships or connections? 

• Can learners identify information that is important to the task, problem, or both? 

• Can learners understand when information or concepts apply? 


Performance 


1. How fluently can learners perform? 




• How much effort is required? 


How well can 


• How consistently do learners start and finish when getting to the desired outcome? 


learners perform, 


• How well are barriers controlled or overcome? 


including both 




fluency and 


2. How independently can the learners perform? 


independence? 


• How much help Is needed from others? 




• How much initiative is shown In getting started? 




• How often do learners generate their own strategies to complete the task? 


Range 


1 . What kinds of tasks do learners carry out? 




• How complex is the task? 


What kind and 


• How many different kinds of tasks can learners perform? 


number of tasks 




can they perform 


2. In what contexts can learners perform? 


and in what 


• In what kinds of circumstances can learners perform? 


context? 


• In how many different situations can learners perform? 



(relative but not absolute) ranges, in part so we could 
look closely at performance in 10-point ranges from 
0-60. By focusing on the 0-60 range, we were able to 
obtain richly detailed descriptions of adult learner 
performance at the levels covered by adult 
literacy/ ABE/GED/ESL programs and described in 
the six ABE and six ESL Educational Functioning 
Levels of the NRS. The descriptions on the template 
were devised as generic “markers” that describe 
increments of growth; they focus on changes in key 
features on each dimension and will be revised, 
expanded, and further specified for each Standard, 
based on teacher documentation. 

A More Comprehensive Picture 
of Learner Abilities 

Teachers tell us that using the EFF Performance Tem- 
plate was initially challenging; its language sometimes 
seemed vague and repetitive (an inevitable conse- 



quence of its generic nature and the incremental 
change that it suggested along the continuum), and it 
took a lot of time to document performance from so 
many “angles.” As they became more experienced in 
using the template, however, they came to appreciate 
how it aUowed them to see a much more comprehen- 
sive picture of their learners’ abilities than they had 
before. They used it as a guide to what prior knowl- 
edge they should take into account as well as what 
new knowledge and skills for which they should 
look — what specific behaviors to identify — when 
they assessed learner performance. Meanwhile, the 
data that they provided by using the template allowed 
us to refine and expand the descriptors in each range. 

Midway through the year, the template was 
restructured in response to feedback from ESL 
teachers. They requested “more room” for rich 
descriptions at the 0-20 ranges in order to take into 
account everything that some ESL learners (particu- 
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larly those who are not literate in their native lan- 
guages) need to learn to accomplish tasks in this 
range. Two new columns were added in the 0-20 
range through all dimensions. These are still relative 
“placeholders” and not absolute ranges, but they are 
meant to allow for more “granular” description at 
lower points on the continuum. 



Definition of an EFF Performance Task: 
Targeting the Standard 

Building the continuum for each Standard required 
reliable data on the performance of each EFF Stan- 
dard. To make sure that activities being documented 
by teachers — researchers squarely focused on the full 
Standard (including all of the Components of Per- 
formance) — we asked teachers to observe and docu- 
ment learner performance of a particular Standard 
in what we termed “well-structured EFF Perfor- 
mance Tasks.” We offered the following definition: “A 
well-structured EFF Performance Task is a learning 
activity that meets learners’ purposes and addresses 
all components of an EFF Standard. It represents a 



real-world, adult use of the targeted Standard that 

can be analyzed according to the four Dimensions of 

Performance.” 

A Set of Criteria for Well-Structured 
Performance Tasks 

In Chapter 3 of the Field Guide, developed for 
teacher/researchers as an orientation 
and preparation for the 2000-2001 
field research, we articulated four cri- 
teria for defining and rating well- 
structured EFF Performance Tasks. 
These criteria, the basis for the work- 
sheet described below, were clarified 
and revised during a midcourse 
meeting in February 2001 of repre- 
sentatives from all field research sites. 

The revised Criteria for a Well- 
Structured Performance Task read as 
follows: 

• The EFF Standards represent the 
knowledge and skills adults need to 
achieve important purposes in their 
lives. A well-structured performance 
task represents one instance of a meaningful use of 
the standard. 

• A well-structured performance task identifies the 
evidence that will be used to determine how well 
the standard was used to carry out the task. 

• A well-structured performance task is defined 
specifically enough so that knowledge-base 
requirements are clear. 

• A well-structured performance task sufficiently 
focuses on the targeted standard and its compo- 
nents of performance so that performance can be 
rated. 

• A well-structured performance task has immediate 
use or high transfer value for the learner. 



''The performance template helps me to know that there s more involved 
than watching them do the task or reading their completed assignment I 
now can identify such areas as how well they perform^ at what rate, and 
with how many mistakes, etc** 

"Looking back into my past teaching years, I realize that I have been 
teaching and assessing real-life activity lessons in the classroom, just not 
under the title of 'Performance Task* What is new is the use of the tem- 
plate or rubrics to do a more holistic assessment. I find this to be a much 
better method of assessment because you see the students abilities in 
many different angles. I still do 'traditional* methods of teaching and 
assessment, such as grammar, pronunciation, spelling, dictation, etc., 
but I see them now as pre-performance-readiness activities.** 
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Figure 3. Teaching and Learning With EFF Standards 



• How well have students learned to 
use the Standard(s) to meet their 
purposes? 

• What can learners now do? 

• What additional practice do 
they need in order to use 
the skill fluently and 
independently 

in a range of 

situations? 



ASSESSING 
LEARNER NEEDS 

An EFF Performance Task 

• Addresses the Standard 

• Provides opportunity for 
students to develop along 
the four dimensions 

• Is Purposeful, Contextual, 
and Constructivist 



* What do learners want or 
need to do? 

• What do learners know and 
what can they do in relation 
to that purpose? 



PLANNING 



• What else do learners need to know in 
order to carry out the learning experience? 

• What Standard(s) do they need/want to focus on? 

• What learning activities can frame/provide a context 
for this purposeful skill development? 

• What underlying skills and knowledge will learners 
need an opportunity to develop and practice? 



As teachers used these criteria to develop and 
implement performance tasks, they found that 
doing so had important implications for how they 
planned and actually carried out instruction, as 
well as for how they observed and documented 
learner performance. Many reported that their 
learners were responding in positive and powerful 
ways to these changes in how they approached 
teaching. For example, as teachers involved learn- 
ers more in planning learning activities, they found 
themselves stepping more into the role of facilitat- 
ing learning. They noted that learners responded 
positively to the opportunity to participate more 
actively in their own learning and to the opportu- 
nities performance tasks afforded to contextualize 
skill development in activities they found to be 
meaningful and important. 



Embedding Assessment in the 
Teaching and Learning Process 

A key learning throughout the process of developing 
and documenting performance tasks was about the 
nature and role of assessment: Who does it, how, and 
when or at what points can we look for evidence of 
learning? Planning and implementing performance 
tasks that were meaningful to learners, rigorous in 
their application of the full Standard, and, therefore, 
effective in producing useful performance data, 
required teachers to think about and “embed” assess- 
ment through all the steps of the planning, teaching, 
and learning cycle. The performance task develop- 
ment process further encouraged teachers to devise 
multiple strategies for eliciting evidence of learning 
at various points in the teaching/learning experience. 
Those were often innovative, performance-based 
strategies that included learner dialogue, 
reflection, and self-monitoring. Based on 
this learning, we developed a new tool for 
teacher/researchers to use as they con- 
structed more performance tasks. This 
new tool is in the form of a graphical rep- 
resentation of how a well-structured per- 
formance task informs all aspects of the 
planning, teaching, and assessing cycle 
(see Figure 3). 



“/ donh spend as much time as before in front of the class 'teaching' 
vocabulary or whatever; the students spend more time now applying 
their learning in real-life applications. I now spend more time and 
energy thinking about and developing tasks that will be meaningful 
for the students. The students seem to understand that important 
indications of their learning happens during their performance of 
these EFF tasks. They want to include these EFF artifacts in their 
portfolios as examples of their best work." 
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**In this learning process, the students were full participants. 

The students were motivated to do the task because the task 
was real to them and relevant to their lives. The students made 
connections between the subject matter and how to use the sub- 
ject in their lives instead of thinking that school is 'stuff' they 
learn in school and will never use again. As a teacher, this type 
of teaching was creative, exciting, and individualistic. I was 
more of a facilitator in the process than usual. It was different 
than the way I usually teach because each student could apply 
the lesson to themselves and their lives." 

**Though ESL teachers inherently know that time is critical for 
mastering language, we get caught up in deadlines and progress 
testing and imposed or perceived curriculum demands. Creat- 
ing a well-structured, step-by-step process for students to get to 
their final writing piece (contrasting past, present, and future 
English learning) made this learning activity easy for the stu- 
dents and for me. Once they got started, it seemed there was no 
stopping them. And yet I worried that it was taking too much 
time. I wondered if the increased fluency that I saw had to do 
with the topic, the process, or both." 

''My teaching has become very focused. The performance task 
has been integrated into the lessons and curriculum, but my 
emphasis is always on the performance of the specific task. 
Starting with the task and working backwards in the lesson- 
planning process is opposite of the way I used to plan and teach. 
Formerly, I would choose a theme or unit (that may or may not 
have been chosen by the students) that I felt was important. 
Then I would develop a scope and sequence. Tasks would be 
teased out along the way as they developed. Emphasis would 
be placed where students needed help. There was never really 
any real product or end performance except for the GED test. 
The EFF Framework has changed my approach. It is useful in 
that most students become involved. They like having a con- 
crete task toward which they are working." 



The EFF Performance 
Task Template 

Teacher/researchers not only developed and 
implemented well-structured EFF Performance 
Tasks, they were also describing and rating the 
“objective difficulty” of those tasks. Rating Per- 
formance Tasks along a continuum of increas- 
ing complexity helped teachers be sure that 

• the proposed tasks were appropriately 
chaUenging (not too easy or too difficult) 
for learners at a given “level,” 

• they could think about what learners 
already knew and could do that would 
contribute to successful completion of the 
proposed tasks, 

• they could define what more the learners 
needed to know and be able to do in 
order to successfully perform the tasks 
and how much or what kinds of “scaf- 
folding” might need to be built in to 
activities, and 

• they would have a sound basis of compar- 
ison for documenting learner perfor- 
mance of the task. 

To assist practitioners in this process, we devel- 
oped a Task Template, adapted from the struc- 
ture, content, and theoretical foundations of 
the Performance Template. The Task Template 
focuses on six ranges of complexity, contexts, 
and knowledge-base requirements for pro- 
posed tasks, and the template supports users in 
richly describing and consistendy rating their 
tasks for these characteristics. In the task exam- 
ple found in Appendix C, the teacher used the 
task template to guide her detailed descriptions 
of the requirements of a task that involved 
reading the classified ads to find affordable and 
suitable housing. 
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“Owr [team] conversations are usually centered on helping one 
another see the relative difficulty of the task and rating it on the Task 
Template, based only on the task's difficulty rather than on how well 
our particular students may be able to perform. Having the other 
ESL, Even Start, and ABE instructors involved in our group is useful 
in allowing us to bounce ideas around and to ensure that the task is 
being rated on the difficulty for all students, not just our own," 

The worksheet for developing a well- structured performance task is 
very helpful — but I work backwards through it. By the time I finish, I 
have a clear task and better plans for the lessons leading to the task. 
After [the project midcourse meeting] I have a much better idea of 
how to write the tasks. Now that a 'clear, succinct task* is the goal 
I feel I have a better teaching objective and a better means for 
measuring student achievement.** 

'Tn developing this task, the things that were the most helpful 
were integrating the Standard into the task, looking at the student's 
purpose for wanting to improve in this skill area, and working with 
other staff members to develop the 
task. Student interest and need to work 
on the concept of cooperation in the 
worker role acted as a guide for devel- 
oping this task." 



tasks, and to provide a way to keep track 
of their notes, a worksheet (Figure 4) was 
developed that mirrored the questions 
found on the Data Reporting Form, with 
additional prompts to make sure that the 
task was fully analyzed and rated. 

During the first reporting period 
(October-December 2001), teachers for- 
warded a completed worksheet to their 
field assistant, who reviewed the task 
analysis and gave feedback on ways to 
sharpen the focus on the standard. Later, 
with more experience under their belts, 
teachers did not need to use this work- 
sheet; but many continued to do so, find- 
ing it to be a useful way to develop a 
complete and well-structured Perfor- 
mance Task. They noted that using the 
worksheet helped them feel surer that the 
tasks they developed were appropriate to 



Performance Task 
Worksheet: Targeting 
Instruction to Student Need 

As can be imagined, the process of 
going back and forth between the 
Standard and the Dimensions of 
Performance as described in the 
generic task template in order to 
develop a well-structured Perfor- 
mance Task was complicated. To 
help teachers think clearly about the 
structural requirements of their 



Figure 4. Performance Task Worksheet. 
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the needs and skill levels of their 
learners. Further, the focus on care- 
ful planning that using the work- 
sheet required paid off in more 
sharply focused, better organized 
teaching. When teachers/researchers 
turned to documenting student 
performance of tasks, we added 
another set of questions to the Per- 
formance Task Worksheet to pro- 
vide the same opportunity to think 
clearly about the performance that 
the task may elicit and the evidence 
that the teacher would use to docu- 
ment student performance. 



Figure 5. Teacher Observation Form 
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**The difficult part is trying to make sure that all of the prior 
knowledge^ vocabularyy and understanding are there in order to 
accomplish the task. The most challenging part for me is to create a 
task that is not too difficult for my beginning literacy students^ yet 
challenging for the high-level students who are all in the same 
class** 

**My teaching becomes very focused when i*m doing the tasks. The 
whole process of developing the task carries over to lesson planning 
and helps me to be more organized. It*s not that / wasn*t organized 
before, it*sjust that the focusing is easier. The 'performance task* 
way of teaching is starting to appear spontaneously in my teaching, 
even teaching that is not specifically related to the performance 
tasks for this project.** 



Teacher Observation and 
Student Documentation Forms 

As part of the initial training and support offered to 
participants in the 1999-2000 phase of this field 
development process, we outlined detailed, step-by- 
step procedures for data collection and reporting. 
Since we knew the burden of observing and docu- 
menting performance would be considerable, we 



urged teachers to build plenty of time into 
their instructional activities for them to 
reflect with their learners on skills being 
developed and tasks being accomplished in 
the teaching/learning process. We also 
encouraged them to set aside time at least 
once each week to document growth and 
change in extensive detail so that the result- 
ing data would be useful in building a con- 
tinuum of performance. 

To support the process of teacher obser- 
vation and student reflection, we developed 
a tool that focuses teachers* written obser- 
vations around the dimension-related ques- 
tions found on the generic Performance 
Continuum (which, themselves, approxi- 
mated a set of observation protocols). The EFF 
Performance Observation Worksheet (Figure 5) 
structured teacher observations of performance as 
answers to four questions aligned to the four 
Dimensions of Performance: 

1. What kinds of tasks can learners carry out 
(range)? 

2. In what contexts can learners perform (range)? 

3. What do learners know (knowledge base)? 
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4. How well can learners perform (flu- 
ency/independence)? 

In response to teacher requests, a 
shortened version of the form was made 
available for students to use to docu- 
ment their own learning (Figure 6). 

This tool proved to be a valued 
resource to many teachers for diagnos- 
tic and instructional purposes, and it 
was used again during 2000-2001 data 
collection. It helped teachers to orga- 
nize their observation notes in a way 
that made it easier to compile relevant 
information over time and then trans- - 

fer that information to the Teacher 
Observation Form. Since the forms were complet- 
ed at the time of observation or shortly thereafter, 



'^Planning and teaching for EFF performance tasks is enjoyable: 

I like the process, and the students seem to like it, as well Observing 
student performance is one thing, but recording it is something that 
is going to take more practice, and it is very difficult to accomplish 
during the actual student performance. There are so many things 
going on during class that it may be helpful to have someone else 
do an objective recording of the process. Having a video or audio 
recording when evaluating performance would be helpful Providing 
some free time immediately after each performance would also 
allow for some reflection and review of the process, an opportunity 
to fill in voids, and enable a more accurate evaluation of student 
performance. However, Tm not sure how these needs could be met."' 



we found that they provided a great source of rich 
and detailed descriptions of performance. 



Figure 6. Student Documentation Form 
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Implications of Year-One Work 
for Enriching the Educational 
Functioning Levels of the 
National Reporting System 



D uring Year One, the EFF/NRS 
Data Collection Project conducted 
extensive field research that result- 
ed in draft continua of perfor- 
mance for five EFF Standards: 
Read With Understandingy Convey Ideas in Writingy 
Speak so Others Can Understandy Listen Actively, and 
Use Math to Communicate and Solve Problems. In 
addition, teacher/researchers collected performance 
data for five additional Standards: Use Information 
and Communications Technologyy Learn Through 
Researchy Take Responsibility for Learning, Solve Prob- 
lems and Make DecisionSy and Cooperate With Others. 
Specifically, the Project 

• trained more than 100 teachers in 5 states in devel- 
oping instruction and assessing performance on 
the EFF Standards, 

• developed a set of instructional and documenta- 
tion tools that help teachers embed assessment in 
on-going instruction, 

• collected more than 300 teacher-generated perfor- 
mance tasks for 10 EFF Standards, and 
• analyzed performance tasks and descriptions of 
adult learner performances on these tasks to create 
draft performance continua for 5 Standards. 

The data collected by the field researchers, 
using the tools described in the previous section, 
helped define the constructs and rich level descrip- 



tors that contribute to the completion of the EFF 
Assessment Framework, which will include 

• a developmental sequence of descriptions of 
learner performance on the Knowledge Base, Flu- 
ency, Independence, and Range dimensions of the 
EFF Performance Continuum for each of the 16 
EFF Standards that can be used to guide learning 
and instruction; 

• model performance-based assessments and scor- 
ing guidelines (rubrics), based on the Perfor- 
mance Continuum for each EFF Standard, that 
can be used to mark transitions from one level of 
performance to the next (for at least six specified 
levels to correspond to the current six ABE/ASE 
and six ESL NRS Educational Functioning Lev- 
els); and 

• materials, training, and technical assistance to sup- 
port the implementation of these EFF-based cur- 
riculum and instructional resources and assessment 
tools. 

These elements of the EFF Assessment Frame- 
work provide a basis for enriching the NRS Educa- 
tional Functioning Levels and for supporting valid 
and reliable measures of educational gain, using 
standardized alternative assessments by specifying 
characteristics of assessment tasks that can be used 
to collect and evaluate evidence of adult perfor- 
mance at various levels on the EFF Standards. 
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Defining Levels of Performance 

As in any educational assessment, the number of 
levels of performance specified and the definition 
of boundaries between levels is a matter of subjec- 
tive judgment and consensus. There is no “mea- 
surement magic” that can reveal the “true” levels of 
adult performance. However, this does not mean 
that levels of performance on the EFF Standards 
cannot be marked in an empirically grounded and 
procedurally rigorous way. Through careful and 
recursive analysis of field- generated learner perfor- 
mance data and broad-based field and content 
expert review of level descriptions resulting from 
this analysis, we can construct a set of level descrip- 
tors for adult performance on the EFF Standards 
that can, in turn, be used to guide the development 
of assessment tools and scoring guidelines for 
reporting educational gains in the NRS. In this way, 
the EFF Assessment Framework will extend the 
range of assessment options available to adult edu- 
cation programs in measuring and reporting edu- 
cational gains in the NRS. 

The levels of adult performance that are 
marked out in the EFF Assessment Framework will 
be determined on both empirical/theoretical and 
practical grounds. The empirical/theoretical bases 
for determining levels of performance (and for 
developing specifications for performance assess- 
ments and scoring guidelines to benchmark levels of 
performance) consist of the field-based research and 
development processes described in the earlier sec- 
tions of this report plus the content expert review, 
stakeholder review, and validation processes being 
conducted by the EFF Assessment Consortium. The 
practical grounds for determining levels of adult 
performance on the EFF Standards has its basis in 
the conventional system of levels of instruction into 
which students in the adult education and literacy 



system are placed. The NRS levels (6 for ABE/ASE 
and 6 for ESL) reflect this conventional and institu- 
tional division of levels of achievement for students 
in the adult education and literacy system. The data 
gathered through the EFF/NRS Data Collection 
Project has come from adult education programs 
and from students who are classified within levels 
(courses of instruction) that correspond to the NRS 
Educational Functioning Levels. 

By looking carefully at the characteristics of 
performance tasks and at the range of learner perfor- 
mances on tasks by adult students at different pro- 
gram levels corresponding to the NRS levels, we can 
create a rich picture of stages of learning and devel- 
opment on the EFF Standards that can be mapped 
onto the conventional levels of achievement reflected 
in student placement and in the NRS Educational 
Functioning Levels. In this way, we hope to provide a 
nuanced and explicit framework for development 
and selection of assessment tasks and for scoring and 
reporting student performance on the EFF Stan- 
dards that will be instructionally relevant and, at the 
same time, reliable and generalizable enough 
(because it is linked to an explicit continuum of 
adult performance and to clear specifications for 
tasks and for scoring performance at points along 
the continuum) for use for high-stakes program 
improvement and accountability purposes. 

Strengthening Instruction, 

Accountability, and Continuous 
Improvement 

The EFF/NRS Data Collection Project has also creat- 
ed a strong foundation for adult education system 
reform and improvement. The combination of pro- 
fessional development for a cadre of adult educators 
(who can, in turn, train others) and the development 
of tools (performance tasks and performance con- 
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Practitioners at EFF field research sites say that using EFF 

Performance Tasks to document student performance is 

influencing what happens in the classroom. Administrators 

and teachers tell us that teachers are 

• thinking more about students’ prior knowledge — what they bring 
to the learning environment from their previous experience — and 
planning in ways that will build on this prior knowledge; 

• engaging students in thinking about their own learning process- 
es — thinking about thinking — or metacognition; 

• discussing cognitive and metacognitive strategies and their 
(teachers’) need for more professional development in this area; 

• thinking ahead in their lesson planning (What evidence of 
student performance do I expect to see from this activity? How 
will it be documented? What is the role of my students in help- 
ing develop the criteria for the evidence?); 

• continually asking oneself if the activity is transferable and if it 
has real-life meaning; and 

• focusing on the teaching process rather than on the product — 
or, put another way, facilitating student learning rather than 
teaching the “right answers.” 

(Summarized from Ohio teacher reports, 2000-2001.) 



tinua) that facilitate use of assessment 
data in monitoring and improving 
instructional practices has strengthened 
the capacity of adult education systems 
for accountability and continuous 
improvement. When teachers recognize 
the value of monitoring adult student 
learning, when they have appropriate 
and practical tools to monitor learning 
outcomes, when they are able to make 
use of learner performance data to 
improve instruction, and when they can 
accurately assess student progress and 
report learning gains, the promise of 
accountability leading to educational 
improvement can be realized. 

Work on the development of the 
EFF Assessment Framework has been a 
complex and challenging endeavor. We 
have found that our original timeline 
for developing performance continua 
for the 16 EFF Standards has had to be 
revised as we identified more clearly the 
research, analysis, and validation need- 
ed to produce quality results. At the same time, we 
have become more aware of the value of the practical 
tools and professional development provided in the 
course of our development work. Through our expe- 
rience in working with teacher/researchers in devel- 
oping assessment tools and in providing professional 
development in assessment, we have seen that teach- 
ers have a better understanding of evidence and of 
the conditions for assessment, of the big picture of 



assessment, resulting in a group of teachers who are 
better equipped to provide reliable data on learning 
outcomes. These interim results are critical to meet- 
ing the broadest goals of the EFF initiative as well as 
the NRS: continuous improvement of the adult edu- 
cation and literacy system. With continued commit- 
ment from our field partners and increased interest 
from other states, we are looking forward to seeing 
this work through to completion. 
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Appendix A 

Background Documents 



EFF/NRS Data Collection Project Timetable, 2000-2001 
EFF Assessment Framework: Timetable for Processes and Products, 2000-2004 
EFF Assessment Consortium Staff, Technical Advisory Group and Field Development Partners 
Participant Responsibilities and Agreement Forms 
Educational Functioning Level Descriptors, National Reporting System 
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EFF/NRS Data Collection Project Timetable 2000-01 



DATE 


ACTIVITY 


DESCRIPTION OF ACCOMPLISHMENTS 


August 2000 


Letters and applications sent to 
state directors 


Ohio, Oregon, Maine, Tennessee and Washington 
committed funds and began to identify from 
3 to 6 programs per state to participate. 


September 


1 . Sites identified 

2. State field assistants identified 

3. Field site training schedule 
established 

4. Training materials and data 
collection protocols prepared 


1. 20 sites, 20 administrators, 88 teachers 

2. Maine: Janet Smith; Ohio: Kathy Petrek and Sharon 
Katterheinrich; Oregon: Joan Benz; Tennessee: Aaron 
Kohring; Washington: Joan Allen. 

3. In-state training sessions scheduled for October and 
early November. 

4. Guide to the EFF/NRS Data Collection Project 
prepared. Notebook included four chapters on 
developing and rating performance tasks and 
observing and documenting student performance; 
data reporting forms, worksheets and templates 
for rating tasks and performance. 


October 


Five 2-day training sessions 
conducted 

• Oct 9-11: Washington 
•Oct 11-13: Oregon 
•Oct 18-20: Ohio 

• Oct 24-26: Tennessee 
•Oct 31 -Nov 2: Maine 


• In each state, participants included all site teachers 
and program administrators; representatives from state 
adult education agency staff. 

• Training was conducted by Peggy McGuire and 
Brenda Bell, with assistance from the state field 
assistant, following a common agenda that focused 
on constructing performance tasks, as the vehicle for 
observing and documenting student performance. 


November 


Data collection and technical 
assistance process started 


Practitioner-researchers completed preliminary 
worksheets for performance tasks; submitted to 
field assistants for review; received feedback. 


December 


Continued technical assistance 

End of 1st reporting period, 
December 31 


• Field assistants visited each site or held conference 
calls with the research team at each site. 

• First round of data reports on performance tasks 
submitted electronically. 128 reports received on 

10 standards. 20 reports received from administrators, 
describing the impacts of participation in this project 
on the team and program at large. 


January 2001 


Data review 


EFF Assessment Team met to review data reports; 
identify needed corrections to protocols; and plan 
national mid-course meeting content. 


February 


National mid-course meeting, 
February 4-6 

I 


• Over 100 teachers, administrators and state agency 
staff met in Washington, DC for three days to examine 
data on performance tasks, refine criteria for well- 
structured tasks, and prepare for submitting data on 
learner performance. 

• Based on recommendations of participants, staff 
prepared a series of memos and revised the data 
collection form. 

t 
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DATE 


j ACTIVITY 


DESCRIPTION OF ACCOMPLISHMENTS 


February 2001 , 
cont. 


i Technical assistance to the field 
j assistants 


• Staff provided on-going education and technical assis- 
tants to the practitioners who are serving as field assis- 
tants. This process began In October and is on-going. 


February-March 


\ Observations of student 
j performance 


• Using performance tasks developed specifically to 
allow observation of performance on a standard in 
relation to the four dimensions of performance, 
teachers began documenting student performance.** 

• Field assistants held regular telephone or in-person 
meetings with program teams. 


April 


i 1. In-state technical assistance 
meetings held 


1 . Prior to the end of the second reporting period, in- 
state joint meetings of all field sites were conducted 
by the field assistants, with participation by McGuire 
or Bell, to provide additional technical assistance and 
clarification on reporting procedures. 




I 2. End of second reporting period 


2. From mid April to the end of the month, the second 
round of reports were submitted, accompanied by 
artifacts showing evidence of student performance. 
1 1 0 data reports received. (Several teachers have 
either moved or not involved due to health or life 
changes). 


April-May 


1 Technical assistance continued 


Project staff and technical advisors began a series 
of data review telephone conferences, to identify 
strengths and potential problems In the data, and to 
provide guidance to field assistants in giving feedback to 
practitioner-researchers. Regular telephone conferences 
with field assistants were held, to review the same data 
sets. Field assistants held regular telephone or in-person 
visits with field site teams. 


June 


j Data Analysis 


Data from first two reporting periods prepared for use at 
July data analysis meeting. 


July-August 


1 Data analysis and construction of 
j draft continua of performance for 
j these standards: Read with 
i Understanding, Convey ideas in 
j Writing, Listen Actively and Speak 
1 So Others Can Understand 


The EFF Assessment Consortium and field assistants 
met July 9-13, to begin the continua construction 
process, using data from field reports. The team prac- 
ticed the behavioral anchoring process that will be used 
by the technical judging panels. Work continued through 
the end of August to finalize the standard-specific data 
templates and draft continua. Materials were prepared 
for review at the final project meeting. 


September 


j Final project meeting, September 
! 12-15, Portland, Oregon 


Participants reviewed and revised the draft continua for 
four communications standards; evaluated the prelimi- 
nary draft continua for the other six Standards and made 
recommendations for additional rounds of field work. 




32 



34 




EFF/NRS DATA COLLECTION PROJECT. 2000-2001 



EFF Assessment Framework 



Timetable for Process and Products 2000-04 



DATE 


PROCESS 


PRODUCTS 


2000 


Field Research to Describe 
Performance on the EFF Standards 

• Define 4 dimensions of the EFF 
performance continuum 

• Generate performance tasks and collect 
data on adult learner performance 


• 4 dimensions of the EFF performance 
continuum defined. 

• Preliminary picture of performance along 
each dimension of the continuum for 
ABE and ESL learners. 


2001 


Field Review and Analysis to Develop 

EFF Performance Continue 

• Develop/refine performance continue 
for four Communication Standards. 

• Develop/refine performance continuum 
for Use Math. 

• Continue research to generate more 
data for remaining Standards. 


• Draft performance level descriptors 
for each of the 4 EFF Communication 
Standards and for Use Math to 
Communicate and Solve Problems. 

• Range of performance tasks for all 
5 Standards. 

• Practitioners with increased expertise 
in standards-based teaching and 
assessment. 


2002 


Expert Review of Performance 

Continue (Phase 1) 

• Panels of experts review performance 
continue for Listen Actively, Speak So 
Others Can Understand, and Convey 
Ideas in Writing 

• Panels of experts review performance 
continue for Read With Understanding 
and Use Math to Solve Problems and 
Communicate 


• Revised performance level descriptors 
for the knowledge base, fluency, 
independence, and range dimensions 
of the performance continue for these 
3 Standards 

• Revised performance level descriptors 
for these 2 Standards 




Match EFF Performance Continue to 
NRS Levels 

• Use results of content expert reviews of 
the 4 Communication Standards plus 
Use Math to draft EFF/NRS Educational 
Functioning Levels 


• Draft EFF/NRS Educational Functioning 
Level Descriptors for 5 EFF Standards 


November 


Conduct Accountability Model 

Development Workshop 

• Review EFF/NRS Level Descriptors and 
identify benchmarks for use in state 
assessment systems 

• Develop models for state assessment 
and reporting of educational gains using 
the EFF/NRS Level Descriptors 


• Revise EFF/NRS Level Descriptors 

• Draft specifications for assessment task, 
scoring rubrics, and reporting systems 

(continued on page 34) 
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EFF Assessment Framework 



Timetable for Process and Products 2000-04 



DATE 


i PROCESS 


i PRODUCTS 


2002 

continued 


i Plan Phase 2 Expert Reviews 


I • Design for second round of content 
expert review panels 




I Plan Task Development Institute 
I (Phase 1) 


j • Design of Task Development Institute 




i Conduct Task Development Institute 
I (Phase 1) 


i • Assessment tasks, scoring rubrics, 
and reporting guidelines for 4 EFF 
Communication Skills and Use Math 


2003 


j Pilot Test Assessment, Scoring, and 
I Reporting Tools for Phase 1 EFF/NRS 
i Levels 


I • Revisions to assessment tasks, scoring 
rubrics, and reporting guidelines 




i Develop Materials and Resources for 
i Training and Technical Assistance 


I • Handbook and technical assistance 
resources 




I Expert Review of Performance 
i Continue (Phase 2) 


i • Revised performance continue for 

selected Interpersonal, Decision-Making, 
and Lifelong Learning Skills Standards 




I Distribute EFF Tools for Assessing 
i and Reporting Education Gains on 
I EFF/NRS Levels 


i • Specifications for assessment tasks, 
scoring rubrics, and reporting 
I • Sample assessment tasks, scoring 
guidelines, and reporting forms 




i Task Development Institute (Phase 2) 


i • Assessment tasks, scoring rubrics, 
and reporting guidelines for selected 
Interpersonal, Decision-Making, and 
Lifelong Learning Skills Standards 




j Pilot Test Assessment, Scoring, and 
i Reporting Tools for Phase 2 EFF/NRS 
! Levels 

i Update Materials and Resources for 
! Training and Technical Assistance 


i • Revisions to assessment tasks, scoring 
rubrics, and reporting guidelines 
i • Revised handbook and technical 
assistance resources 


2004 

January 


j Distribute Phase 2 Tools 


i • Specifications 

i • Sample tools and support materials 
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Equipped for the Future Assessment Consortium 



Technical Advisory Group 

Sri Ananda 

Program Director, Assessment 

and Standards Development Services 

WestEd 

Lynda Ginsburg 

Senior Researcher 

National Center on Adult Literacy 

University of Pennsylvania 

Dorry M. Kenyon 

Director, Language Testing Division 
Center for Applied Linguistics 



Stephen Reder 

University Professor and Chair 
Department of Applied Linguistics 
Portland State University 

John Sabatini 

Educational Researcher 
University of Pennsylvania 

Chris Sager 
Senior Researcher 
HumRRO (Human Resources 
Research Organization) 



Equipped for the Future Assessment Consortium Staff 



Center for Literacy Studies 

University of Tennessee 
600 Henley Street, Suite 312 
Knoxville, TN 37996 
865-974-4109 
865-974-3857 (fax) 

Brenda Bell 

Consortium Co-Director 

865-974-6654 

bsbell@utk.edu 

Gail Cope 
Research Associate 
865-974-1225 
gcope@utk.edu 

Aaron Kohring 
Research Associate 
865-974-4258 
akohring@utk.edu 

Peggy McGuire 
Senior Research Associate 
4947 Rubican Street 
Philadelphia, PA 19144 
215-843-8384 (ph/fax) 
mcguirep555@aol.com 



SRI International 

333 Ravenswood Ave 
Menlo Park, CA 94025 
650-859-3375 (Fax) 

Regie Stites, Consortium 

Co-Director 

650-859-3768 

reg ie. st ites@ sri . com 

Melanie Daniels 
Research Analyst 
Tel: 650-859-5805 
melanie.daniels@sri.com 

Nadine Duong 
Nadine.duong@sri.com 

Marilyn Gillespie 
Educational Researcher 
1611 North Kent St 
Arlington, VA 22209 
703-247-8510 (W) 
(Fax)703-247-8493 
gillesple@wdc.sri.com 



National Institute for Literacy 

Sondra Stein, National Director 
Equipped for the Future 
National Institute for Literacy 
1725 I Street, NW, #730 
Washington, DC 20006-2401 
202-233-2025 (office) 
202-233-2050 (fax) 
sstein@nifl.gov 
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Equipped for the Future Field Development Partners, 



MAINE 

Maine Department of Education 

Marcia Cook 

Center for Adult Learning and 
Literacy, U of Maine, Orono 

Mary Schneckenburger 

Field Assistant: Janet Smith 



PROGRAMS 
Dover Foxcroft-Mllo 
Adult Education 

Shirley Wright, Director 
Diane Curran 
Anita Johndro 
Edith Miles 

Franklin County Adult 
Basic Education 

Raymond Therrien, Director 
Susan Kelley 
Janet Smith 

Massabesic Adult 

and Community Education 

Barbara Goodwin, Director 
Keith Dawson 
Michael DeAngelis 

MSAD # 27 Adult Education 

Peter Caron, Director 
Mary Ouellette 

MSAD # 49 Adult and 
Community Education 

Patricia Theriault, Director 
James Chapman 
Alverta Dyar-Goodrich 

Noble Adult and 
Community Education 

Brenda Gagne, Director 
Louise Burns 
Jill Hofmelster 



OHIO 

Ohio Department of Education 

Jim Bowling and Denise Pottmeyer 

Ohio Literacy Resource Center 

Judy Franks 

Center on Education and 
TValnlng for Employment 

Cindy Zengler and Lynn Reese 

Field Assistants: Kathy Petrek 
and Sharon Ketterheinrich 



PROGRAMS 

Canton City Schools ABLE 

Jane Meyer, Coordinator 
Martha Hyland, Coordinator 
Stephanie Reinhart 
Debbie Stowers 
Dana Tomcsak 

Columbiana County 
Career Center 

Michael Morris, Coordinator 
Andrea Copestick 
Laura Joan Wagner 

Hamilton City ABLE 

Kathy Petrek, Coordinator 
Tawna Eubanks 
Sharon Katterheinrich 
Millie Kuth 

Ravenna Even Start 

Odessa Pinkard 
Susana Barba 
Meg Kuyon 
Lory Vild 

South Western City Schools 

Gail Morgan, Administrator 
Candy Bettinger 
Karen HIbbert 
Ruth Knisely 
Sharon Trouten 
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OREGON 

Oregon Department of 
Community Colleges and 
Workforce Development 

Sharlene Walker, Kristen 
Kulongoski and Cathy LIndsley 

Field Assistants: Joan Benz 
(deceased); Mary Foust 



PROGRAMS 
Central Oregon 
Community College 

Janet Rippy, Administrator 
Catherine Lund 
Melissa Potter 
DIcksy Scott 

Chemeketa Community College 

Susan Fish, Administrator 
Kay Gerard 
Monica Salgado 
Virginia Tardaewether 

Clackamas Community Collee 

Rene ZIngarelli 
Linda Durham 
Kathleen Fallon 
Alice Goldstein 

Department of Corrections 

Julie Kopet, Administrator 
Tom Gregson 
Judy Neumann 
Janice Ruhl 

Lane Community College 

Dennis Clark, Administrator 
Mary Foust 
Mary Gilroy 
Cathy Russell 
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Equipped for the Future Field Development Partners, 2000-01 



TENNESSEE 

Department of Labor and 
Workforce Development 

Phil White and Hope Lancaster 


WASHINGTON 

Washington State Board for 

Community Colleges 

Israel Mendoza and Brian Kanes 


Center for Literacy Studies 

Connie White and Jean Stephens 


ABLE Network 

Meg Connelly 
Cynthia Gaede 


Field Assistant: Aaron Kohring 


Field Assistant: Joan Allen 


PROGRAMS 

Greeneville City AE Program 

Kim Gass, Supervisor 
Joyce Hopson 


PROGRAMS 

Bates Technical College 

Jacquie Banks 
Robin Stanton 
Brandi Cockreil 


Knox County Adult Literacy 

Jane Knight, Adult Literacy 
Coordinator 
Ellie Gardner 
Emily McDonald-Littleton 
Mary Norris 


Nancy Gepke 

Big Bend Community College 

Terry Kinzel, Families That Work 
Director 
Sandy Cheek 
Becky Jones 


Putnam County AE Program 

Lynda Breeden 
Kathy Howard 
Mary Jeanne Maples 
Jimmie Webber 


Elizabeth Nelson 
Nancy Villarreal 
Valerie Wade 

Community Colleges of Spokane 

Molly Popchock, Program 
Administrator 
Sabina Herdrich 
Katherine Laise 
Karen Snell 
Marianne Steen 

Seattle Central 
Community College 
Andre Loh, Administrator 
Rebecca Boone 
Colleen Comidy 
Joanna Elizondo 
Josefina Saldin 

Wenatchee Valley 
Community College 
Adrienne Tabar 
Erin Cass 
Paula Jaramillo 
Peter Prehn 
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Agreement Forms - EFF/NRS Project 2001-02 



For Teachers 

I have read the project description and participant responsibilities and I agree to: 

• attend Initial training that will prepare me to plan and describe well-constructed learning tasks; 
document observations of performance; and use the data collection/reporting tools; 

• spend six to ten hours per week In EFF-friendly instructional planning, teaching, and documenting/reporting 
on at least two standards (see my choices below); 

• meet with EFF team members In my program on a regular basis (at least bi-weekly) to collaborate on establishing 
rankings for tasks and performances, and share challenges, ideas, resources and accomplishments; 

• take advantage of opportunities to receive technical assistance from EFF field development staff, including 
site visits and participation In two statewide or regional technical assistance sessions through the year; 

• attend two national meetings of field development partners; 

• submit required reports and documentation In format requested (computer disk) and In a timely manner; 

• ask for help when needed; and 

• inform the project staff promptly If there are any changes In teaching circumstances that prevent or hinder the 
Implementation of the work as outlined above. 

Plans 

I plan to work with the following group of students or class: (please describe educational level 
and type of class or group of students) 

I would like to focus on the following The other Standard(s) I would like 

Communication Standard: to focus on are: 

First choice: First choice: 

Second choice: Second choice: 

Payment for Teachers: I understand that I will receive an honorarium for my participation In this project, and that 
the honorarium amount Is not meant to be a direct reimbursement for each hour spent on work associated with the 
project. I understand that the national EFF management has recommended a stipend of $2,000 to $2,500 per 
teacher and that the final decision about the amount of the honorarium will be made by the state office of adult 
education, which is providing the honorarium. Expenses associated with the Initial In-state training and subsequent 
state meetings of field sites will be covered by the state agency. 

For Program Administrators: 

I have read the project description and participant responsibilities and I agree to: 

• support a team, consisting of three to four instructors and myself. In our Involvement 
In the phase three field development process; 

• assure that members of the team meet regularly and work collaboratively as much as possible 
to accomplish the goals of the EFF Phase 3 field development initiative; 

• attend all training and technical assistance sessions, including state or regional and national meetings; 

• observe Instructional and documentation activities of the teachers; 

• observe the effects of EFF Implementation in my program; 

• convene and actively participate in regular team meetings; 

• encourage other professional development opportunities such as teacher cross- visitation/observation; 

• take advantage of opportunities to receive technical assistance from EFF field development staff 
through site visits and meetings; 

• insure that teacher reports and supporting documentation are completed and submitted on time; 

• submit reports in format requested (computer disk) and in a timely manner. 

• Inform project staff promptly If there are any changes in the program’s ability to participate in the project; 

• manage the grant from the state agency that will provide stipends to participating teachers; and 

• keep state agency contacts Informed about the work. 
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Time Frame 

October, 2000 through June, 2001 , with a national debriefing meeting in September, 2001 ; see attached time table. 

Equipped for the Future, through its grantee, the Center for Literacy Studies at the University of Tennessee, will pay 
for travel, lodging and meals for two national meetings (February and September). 

Agreed: 



Practitioner/Administrator 



Program Director/State Agency Director 



Date 



Date 



Social Security Number 



For State Agency Directors: 

I have read the project description and participant responsibilities and I agree that I 
or my designated representative will: 

• identify four to six programs in this state to participate in this project; 

• commit to providing financial support to each of these field sites for their participation in meetings (two 
statewide site meetings), and for planning and documentation (a stipend of $6,000 - $10,000 per program site, 
or the equivalent of $2,000 - $2,500 per practitioner researcher) 

• ensure that the field sites collectively represent a range of ABE (beginning. Intermediate, ASE) and ESOL 
(beginning. Intermediate, advanced) learners, unless otherwise negotiated with the EFF staff; 

• keep well informed about what is happening in the field development process — by attending all trainings 
and meetings or by sending representatives 

• think about how to integrate what is happening with EFF field development Into the ongoing work of the 
statewide adult basic education system; 

• make sure that three to four instructors and one administrator at each program are actively involved in field 
research/documentation, and are available for initial training, two state-wide/regional technical assistance 
sessions, on-site technical assistance, and two national meeting of pilot programs; and 

• coordinate arrangements for initial in-state training and statewide/regional technical assistance sessions, 
both internally and with the EFF Assessment Coordinator who will also attend the meetings. 

Expenses: Expenses associated with the initial in-state training and subsequent state meetings of field sites 
will be covered by the state agency. Equipped for the Future, through its grantee, the Center for Literacy Studies 
at the University of Tennessee, will pay for travel, lodging and meals for all participants for two national meetings 
(February and September), and technical assistance and support. 



Agreed: 



state Director of Adult Education 



Equipped for the Future Director 



Date 



Date 
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Appendix B 

Assessment, Tools and Templates 



The EFF Purposes of Assessment Chart and Background on its Development 

EFF Performance Template 
Teaching and Learning with EFF Standards 
EFF Task Template 

Worksheet: Developing a Well-Structured Performance Task 
Student Documentation Form 
Teacher Observation Form 
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Background on the Development of the EFF Purposes of Assessment Chart 

Early in the work of developing the EFF Assessment Framework, we identified three primary purposes for 
assessment that the Framework needs to address: 

1. to provide information on learner progress that is useful during the instructional process; 

2. to provide information about learner achievements to stakeholders in the adult education system; 

3. and to provide information that is useful for program and system accountability 

Drawing on the recommendations of the expert review panels for the EFF Content Standards (held in 
January, 1999) during which panelists discussed possible assessments for the standards, we created a draft 
‘‘EFF Purposes of Assessment” matrix by asking, and proposing answers to the broad framing questions: 

• Who needs the results of assessment of adult learners? 

• Why and for what purposes do they need those results? 

• When/at what points in the learning process do they need those results? 

The structure of this matrix was informed by Linking Educational Assessments: Concepts^ Issues^ Methods 
and Prospects hy Robert Mislevy (ETS, 1992) and was based on an EFF-friendly adaptation of a chart that 
appears in that publication {Table 1: Description of Assessment Purposes) which itself was adapted from 
Millman and Greenes Table 8.1 (1989). 

For example, in developing the vertical categories of “who and why/for what purpose,” Mislevy s type of 
inference desired became our “what do we want to learn from assessment?” Description of individual 
examinees* attainments became our “what an individual learner knows/ is able to do to meet self-identified 
purposes in roles roles of family member/worker/citizen.” 

Mislevy’s mastery decision was changed to “Individual Mastery/Credentialing: how much the learner 
knows/how well the learner is able to perform against broad and broadly accepted criteria for ‘what matters’ 
to adults according to their purposes in their roles” and the category description of performance for a group or 
system became 2 categories of “what we want to learn and why: 1) ‘Program Improvement/ Accountability’ 
and 2) ‘System Improvement/ Accountability.’” 

To address the horizontal categories of “when/at what points” we used Mislevy’s Curricular Domain 
(before/during/after instruction); his Cognitive Domain and Future Performance in Criterion Setting became 
our “Uses of assessment results beyond instructional setting” and “Next-Step uses of assessment results.” 

We then filled in the matrix based on our collective knowledge and experience. During the April 
1999 meeting of field sites, participants reviewed the matrix raised issues and concerns, and offered 
suggestions for further work. Later a smaller Assessment Workgroup spent a full day working with the 
matrix in order to identify currently used and/or available assessment instruments and strategies 
that might align with the various purposes (and be useful before, during and/or after instruction); 
suggest other/alternative/new/needed assessment tools to meet the requirements of the EFF Standards 
for the variety of purposes; and provide further guidance on the structure/con tent/ overall usefulness of 
the draft purposes matrix. 
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Template: Documenting Adult Learner Performance 
Against EFF Standards 



EFF Standard: 



Components of Performance: 



1 Dimensions 0-5 


6-10 


11-15 


16-20 


21-30 


31-40 


KNOWLEDGE 

BASE: 

What do 
learners know? 
1. What 
vocabulary 
do learners 
have related 
to the skill? 
Related to the 
subject area? 




Simple 

vocabulary 




Simple 
vocabulary 
with some 
multi-syllabic 
words 


Growing 
vocabulary with 
a good store of 
multisyllabic, 
non-technical 
words 


Moderate store 
of vocabulary, 
some new and 
technical 


2. What 
content 
knowledge 
do learners 
have related 
to the skill? 
Related to the 
subject area? 




Minimal 
familiarity with 
content- 
related facts, 
operations, 
concepts, rules, 
protocols and/ 
or practices 




Familiarity with 
a small store 
of content- 
related facts, 
operations, 
concepts, rules, 
protocols and/ 
or practices 


Familiarity with 
a good enough 
store of facts, 
operations, 
concepts, rules, 
protocols and/ 
or practices to 
carry out the 
task 


Familiarity with 
a good enough 
store of facts, 
operations, 
concepts, rules, 
protocols and/ 
or practices to 
carry out the 
task 



1 Dimensions 


41-50 


51-60 


61-70 


71-80 


81-90 


91-100 


KNOWLEDGE 

BASE: 

What do 
learners know? 
1. What 
vocabulary 
do learners 
have related 
to the skill? 
Related to the 
subject area? 


Moderate store 
of vocabulary, 
some new and 
technical 


Good store of 
vocabulary, 
including some 
new and 
technical 


Good store of 
vocabulary, 
including some 
new and 
technical 


Large store of 
vocabulary 
including new 
and technical 


Extensive 
vocabulary 
that includes 
technical and 
infrequently 
used terms 


Extensive 
vocabulary 
that includes 
technical and 
infrequently 
used terms 


2. What 
content 
knowledge 
do learners 
have related 
to the skill? 
Related to the 
subject area? 


Familiarity with 
a good enough 
store of facts, 
operations, 
concepts, rules, 
protocols and/ 
or practices to 
carry out the 
task 


Familiarity with 
a range of facts, 
operations, 
concepts, rules, 
protocols 
and/or practices 
to meet the 
demands of 
the task 


Familiarity with 
a range of facts, 
operations, 
concepts, rules, 
protocols and/ 
or practices to 
meet the 
demands of 
the task 


Familiarity with 
a range of facts, 
operations, 
concepts, rules, 
protocols and/ 
or practices to 
meet the 
demands of 
the task 


Extensive, 
advanced 
and complex 
content 
knowledge 
useful for 
multiple 
purposes 


Extensive, 
advanced 
and complex 
content 
knowledge 
useful for 
multiple 
purposes 
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EFF/NRS DATA COLLECTION PROJECT. 2 0 0 0 - 2 0 0 1 



Documenting Adult Learner Performance Against EFF Standards 



Dimensions 


0-5 6-10 


11-15 


16-20 


21-30 


31-40 


KNOWLEDGE 




Extremely 




Concrete 


Initial “pattern 


Pattern 


BASE: 




concrete 




activation of 


recognition,” 


recognition, i.e.. 


3. What 




activation of 




prior learning; 


i.e., beginning 


“chunking” and 


strategies do 




prior learning; 




a small number 


“chunking” and 


elaboration of 


learners have 




only a small 




of strategies. 


elaborating of 


information; 


for organizing 




number of 




including 


information; 


activation of 


and appiying 




strategies. 




restatement. 


some early 


prior learning 


content 




limited to 




paraphrase. 


abstraction in 


more abstract 


knowiedge? 




simple recall 




some explana- 


activation of 


and complex; 


•can ieamers 




of Information 




tion; can use 


prior learning; 


some strate- 


recognize 








some examples 


some 


gies, mostly 


reiationships 










strategies: 


simple, some 


or connec- 










explanation. 


“higher-order”; 


tions? 










summary. 


ability to 


• can learners 










paraphrase. 


classify and 


create new 










restatement. 


categorize 


reiationships 










use of examples 


Information; 


or connec- 












some recogni- 


tions? 












tion of cause 



and effect 

relationships; 

explanation, 

interpretation, 

translation, 

some 

generalization. 

Inference, 

prediction 



• can learners 
identify 
information 
that is 
important to 
the task/ 
probiem? 




Little conscious 
ability to identify 
important 
information 




Limited ability 
to identify 
important 
information 


Some 
conscious 
ability to Identify 
information that 
is relevant to a 
clearly-defined 
purpose 


Ability to 
recognize 
and restate 
important 
information for 
application to a 
clearly defined 
purpose 


• can iearners 
understand 
when 

information 
or concepts 
apply? 




Minimal 
understanding 
of when 
to apply 
information 




Limited 
understanding 
of when 
to apply 
information 


Some 

understanding 
of when 
to apply 
information 


Good 

understanding 
of when 
to apply 
Information 
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Documenting Adult Learner Performance Against EFF Standards 



I Dimensions 


41-50 


51-60 


61-70 


71-80 


81-90 


91-100 


KNOWLEDGE 


Pattern 


Pattern 


Beginning 


Some pattern 


A range of 


A broad range 


BASE: 


recognition, i.e., 


recognition, i.e.. 


of pattern 


creation, i.e.. 


complex. 


and variety of 


3. What 


“chunking” and 


“chunking” and 


creation, i.e.. 


“restructuring” 


embedded 


complex. 


strategies do 


eiaboration of 


elaboration of 


“restructuring” 


into new 


strategies for 


embedded 


learners have 


information; 


information; 


into new 


meanings 


integrating 


strategies for 


for organizing 


abiiity to 


activation of 


meanings 


based on 


prior iearning 


integrating 


and applying 


activate prior 


prior knowledge 


based on 


complex 


with new 


prior learning 


content 


iearning in 


and integration 


complex 


integration of 


information; 


with new 


knowledge? 


abstract and 


of new rules/ 


integration of 


prior learning 


ability to 


information; 


• can learners 


compiex ways, 


principles/etc. 


prior learning 


and new 


restructure 


creation of 


recognize 


integrating and 


to solve some 


and new 


information; 


content 


new, multiple 


relationships 


appiying some 


problems; 


information; 


multiple 


knowledge in 


patterns of 


or connec- 


new concepts; 


multiple 


multiple 


strategies 


new ways to 


meaning 


tions? 


range of simpie, 


strategies. 


strategies; 


allowing 


yieid new 


and new 


• can learners 


and a few 


some simple 


comparison/ 


understanding 


meanings and 


organizational 


create new 


"higher order" 


and some 


contrast. 


of both content 


new systems of 


structures; 


relationships 


strategies; 


“higher order”; 


analogies. 


and form 


understanding. 


proposing/ 


or connec- 


identification 


identification 


relationships 


(organizational 


Bias recogni- 


developing 


tions? 


of cause and 


of cause 


between 


structure/ 


tion, criticism. 


alternate 




effect reiation- 


and effect 


concepts and 


relationship of 


conclusion. 


systems of 




ships; some 
generalization, 
inference, 
prediction 


relationships; 

some 

generalization, 

inference, 

prediction 


related details; 

inference, 

prediction 


parts to each 

other). Analysis, 

generalization, 

inference, 

prediction, 

abstraction. 


Justification. 


knowledge and 

understanding; 

consultation 

with multiple, 

alternative 

sources of 

Information 


•can learners 


Abiiity to 


Ability to 


Ability to 


Ability to 


Conscious 


Consistently 


identify 


identify 


identify new 


identify 


identify 


identification of 


“conditioned” 


information 


important 


information and 


relevance of 


relevance of 


important/ 


knowledge; 


that is 


Information for 


to self-monitor 


information 


information 


relevant 


elimination of 


important to 
the task/ 
problem? 


appiication 


comprehension 


for multiple 
purposes 


for multiple 
purposes 


information 
for multiple 
purposes in 
a variety of 
contexts 


incorrect/ 
irrelevant 
information; 
strategic 
adaptation or 
“tuning” of skill 
processes for 
particular uses 


• can learners 


Good 


Strong 


Broad 


Broad 


Broad 


Broad 


understand 


understanding 


understanding 


understanding 


understanding 


understanding 


under-standing 


when 


of when to 


of when 


of when 


of when/under 


of when/under 


of when/under 


information 


apply 


to apply 


to apply 


what conditions 


what conditions 


what conditions 


or concepts 
apply? 


Information 


information 


information 


to apply 
information: 
ability to 
choose best 
option among 
several 
possibilities 


to apply 
information; 
abiiity to 
choose best 
option among 
several 
possibilities 


to apply 
information; 
ability to 
choose best 
option among 
several 
possibilities 




51 

r o BEST COPY AVAILABLE 

be 




EFF/NRS DATA COLLECTION PROJECT, 2000-2001 



Documenting Adult Learner Performance Against EFF Standards 



I Dimensions 0-5 


6-10 


11-15 


16-20 


21-30 


31-40 


PERFORMANCE: | 
How well 
can learners 
perform: 

1. How fluently j 
can learners 
perform? 

• How much 
effort is 
required? 


Performs slowly, 
with difficulty, 
requiring great 
effort 




Performs slowly, 
with difficulty, 
requiring great 
effort 


Performs slowly, 
with noticeable 
effort (or 
inappropriately 
quickly, with 
Insufficient 
attention) 


Performs with 
some hesitation 
but with more 
appropriate 
speed and more 
comfort 


• How 

consistently 
do learners 
start and 
finish, getting | 
to the desired j 
outcome? 


Makes a lot 
of errors, 
produces little 
and has a hard 
time finishing 




Makes a lot 
of errors, 
produces little 
and has a hard 
time finishing 


Work is 

completed with 

considerable 

errors 


Work is 

completed with 
some errors 


• How well 
are barriers 
controlled or i 
overcome? 


Is easily 
diverted/ 
defeated by 
barriers 




Is easily 
diverted/ 
defeated by 
barriers 


Can identify 
some barriers 
but has a hard 
time controlling/ 
overcoming 
them 


Can identify 
barriers and 
possible 
options for 
controlling or 
overcoming 
them; can 
pursue some 
options 


2. How inde- 
pendently can 
the learners 
perform? 

• How much 
help Is 

needed from j 
others? 


Needs 

substantial help 
from others 




Needs 

substantial help 
from others 


Needs 

substantial help 
from others 


Needs some 
help from others 


• How much 
initiative is 
shown in 
getting 
started? 


Needs to be 
“pushed” to get 
started 




Needs to be 
“pushed” to get 
started 


Needs 

considerable 

prompting 


Needs some 
prompting 
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Documenting Adult Learner Performance Against EFF Standards 



I Dimensions 


41-50 


51-60 


61-70 


71-80 


81-90 


91-100 


PERFORMANCE: 
How well 
can learners 
perform: 

1. How fluently 
can learners 
perform? 

• How much 
effort Is 
required? 


Performs at a 
pace sufficient 
to finish, with 
growing comfort 


i Performs with 
i ease; pace may 
i be measured for 
i thoroughness 


Performs with 
ease; pace may 
be measured 


Performs with 
ease and speed 


Performs 
effortlessly, 
smoothly in 
well-organized 
steps, quickly 


Performs 
effortlessly, 
quickly and 
automatically 


• How 
consistently 
do learners 
start and 
finish, getting 
to the desired 
outcome? 


Work is 

completed with 
some errors 


j Work Is 
j completed with 
j few errors 


Work is 

completed with 
few errors 


Work is com- 
pleted with few 
errors 


Work is 

consistent, fully 
completed and 
almost error- 
free 


Work Is 

consistent, fully 
completed and 
almost error- 
free 


• How well 
are barriers 
controlled or 
overcome? 


Can strategize 
about how to 
address barriers 
and pursue 
options to 
control/ 

overcome them 


1 Can strategize 
1 about how to 
j address barriers 
j and pursue 
j options to 
j control/ 
i overcome them 


Can strategize 
about how to 
address barriers 
and pursue 
options to 
control/ 

overcome them 


Controls/ 
overcomes 
most barriers 


Regularly 
addresses/ 
overcomes 
barriers as they 
arise 


Regularly 
engages in 
complex 
processes and 
address/ 
overcomes any 
barriers that 
arise from them 


2. How inde- 
pendently can 
the learners 
perform? 

• How much 
help Is 
needed from 
others? 


Needs some 
help from others 


1 May need some 
\ help from others 


May need some 
help from others 


Rarely needs 
help from others 


Needs no help 
from others 


Needs no help 
from others 


• How much 
initiative is 
shown In 
getting 
started? 


Gets started 

without 

prompting 


1 Gets started 
j without 
j prompting 


Gets started 

without 

prompting 


Gets started 
without 

prompting; may 
initiate new 
learning 
activities 


Gets started 
and Initiates 
activities 
without 
prompting 


Initiates 
activities and 
creates 
new learning 
activities 
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Documenting Adult Learner Performance Against EFF Standards 



I Dimensions 0-5 


6-10 


11-15 


16-20 


21-30 


31-40 1 


PERFORMANCE: ! 

• How Often 
do learners 
generate 
their own 
strategies 
to complete 
task? 


Depends 
upon outside 
structures, 
approaches, 
clarification, 
strong guidance 




Depends 
upon outside 
structures, 
approaches, 
clarification, 
strong guidance 


i Needs 
j significant 
i structures, 
i approaches, 
j clarification, 

I guidance 


Needs 

structures, 

approaches, 

clarification, 

guidance 


RANGE: 

1. What kinds 
of tasks did 
learners carry 
out? 

• How complex j 
Is the task? 


Simple, 
one-step, 
well defined 
and highly 
structured, 
requiring limited 
prediction or 
judgment 




Simple, finite 
but can be 
more than 
one step, 
well-defined 
and highly 
structured, 
requiring limited 
prediction or 
judgment 


j Simple, more 
i than one step, 

: well defined 
j and highly 
: structured, 
j requiring some 
j prediction or 
i judgment 


Multi-step, 
requiring 
integration of 
more than one 
skill; definition 
and structure 
provided; 
requires some 
prediction and 
judgment 


• How many 
different 
kinds of 
tasks can 
learners 
perform? 


Single task 




Single task 


j More than one 
j task 


More than one 
task 


2. In what 
contexts can 
learners 
perform? 

• In what kinds j 
of contexts? 


Familiar 




Familiar 


j Familiar 


Some familiar 
and some novel 


• In how many j 
different 
situations 
can learners 
perform? 


Single situation 




Single situation 


i More than one 
i situation 


More than 
one situation, 
indicating some 
“near” skill 
transfer, i.e., 
into similar 
situations 
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Documenting Adult Learner Performance Against EFF Standards 



Dimensions 41-50 51-60 61-70 71-80 81-90 91-100 



PERFORMANCE; 

• How often 
do learners 
generate 
their own 
strategies 
to complete 
task? 



Needs limited 
structures and 
guidance; can 
generate some 
strategies on 
own 



Needs limited 
structures and 
guidance; 
can generate 
strategies 
on own 



Needs limited 
structures and 
guidance; 
can generate 
strategies 
on own 



Can generate 
strategies on 
own; shows 
some ability to 
be adaptive 
and flexible in 
problem-solving 



Generates 
multiple 
strategies on 
own without 
need of 
structure or 
guidance; can 
choose best 
option; adaptive 
and flexible in 
problem-solving 



Generates 
multiple 
strategies 
including 
consultation 
with outside 
sources of 
information; 
approaches 
tasks without 
need of 
structure or 
guidance; can 
explain tasks to 
others and offer 
guidance; can 
choose and 
justify the most 
appropriate 
approach; 
highly adaptive 
and flexible in 
problem solving 



RANGE: 

1. What kinds 
of tasks did 
learners carry 
out? 

• How complex 
is the task? 


1 Multi-step, 

1 requiring 
I integration of 
I skills and prior 
1 knowledge; 

\ some definition 
j and structure 
1 provided; 
i requires some 
\ prediction and 
j judgment 


Multi-step, 
requiring 
integration of 
many skills 
and prior 
knowledge; little 
definition and 
structure 
provided; 
requires 
prediction and 
judgment 


Multi-step, 
requiring 
integration of 
skills and prior 
knowledge; little 
definition or 
structure; 
requires 
prediction 
and judgment 


Multi-step, 
requiring 
integration of 
skills and prior 
knowledge; 
no obvious 
definition or 
structure 
provided; 
requires 
prediction and 
judgment 


Complex 
tasks featuring 
multiple, 
integrated steps 
and requiring 
frequent 
prediction and 
judgment 


Complex tasks 
with multiple. 
Integrated 
steps; self- 
initiated/ 
self-defined 
tasks requiring 
frequent 
prediction and 
judgment 


• How many 


i Multiple tasks 


Multiple tasks 


Multiple tasks 


Large number 


Large number 


Wide range and 


different 








of tasks 


of tasks 


variety of tasks 


kinds of 














tasks can 














learners 














perform? 














2. In what 


i Some familiar 


Some familiar 


Some familiar 


Familiar and 


Little distinction 


Little distinction 


contexts can 


: and some novel 


and some novel 


and some novel 


novel 


in performance 


in performance 


learners 










between familiar 


between familiar 


perform? 










and novel 


and novel 


• In what kinds 














of contexts? 














• In how many 


j Multiple 


Multiple 


Multiple 


Multiple 


Systematic 


Systematic 


different 


1 situations, 


situations. 


situations. 


situations. 


transfer across 


“near” and “far” 


situations 


I indicating some 


indicating some 


indicating some 


with consistent 


a large range of 


transfer of skill 


can learners 


j “near” skill 


skill transfer 


skill transfer 


transfer to 


“near” and “far” 


across multiple. 


perform? 


I transfer, i.e.. 


into similar and 


into similar and 


“near” and 


(I.e., novel, diffi- 


varied, complex 




I into similar 


some novel 


some novel 


novel situations 


cult, complex) 


environments 




I situations 


situations 


situations 




contexts 
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Teaching and Learning With EFF Standards 



• How well have students learned to 
use the Standard(s) to meet their 
purposes? 

• What can learners now do? 

• What additional practice do 
they need in order to use 
the skill fluently and 
Independently 
in a range of 

situations? TEACHING 



ASSESSING 
LEARNER NEEDS 

An EFF Performance Task 

• Addresses the Standard 

• Provides opportunity for 
students to develop along 
the four dimensions 

• Is Purposeful, Contextual, 
and Constructivist 



What do learners want or 
need to do? 

What do learners know and 
what can they do in relation 
to that purpose? 



PLANNING 



• What else do learners need to know in 
order to carry out the learning experience? 

• What Standard(s) do they need/want to focus on? 

• What learning activities can frame/provide a context 
for this purposeful skill development? 

• What underlying skills and knowledge will learners 
need an opportunity to develop and practice? 
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Template: Analysis of Performance Task 
Requiring Effective Use of an EFF Standard 



EFF Standard: 



Components of Performance: 



1 Dimensions 


0-10 


11-20 


21-30 


31-40 41-50 


51-60 


COMPLEXITY: 


Simple, 


I Simple, finite 


i Simple, more 


Multi-step i Multi-step 


Multi-step 


I.How 


one-step, 


\ but can be 


1 than one step. 


task requiring I task requiring 


task requiring 


complex is 


well defined 


I more than one 


j well defined 


integration of j integration of 


integration of 


the task? 


and highly 


I step, well 


\ and highly 


more than one \ skills and prior 


many skills 




structured; 


: defined 


i structured; 


skill; definition/ i knowledge; 


and prior 




requires limited 


I and highly 


: requires some 


structure pro- i some definition/ 


knowledge; 




prediction or 


! structured; 


! prediction or 


vided; requires ! structure 


little definition/ 




judgment 


I requires limited 


\ judgment 


some prediction i provided; 


structure pro- 






i prediction or 




and judgment j requires some 


vided; requires 






: judgment 




I prediction and 


prediction and 










j judgment 


judgment 


CONTEXTS: 












In what 












context(s) 












will the task 












be performed? 












1. How familiar 


Familiar 


j Familiar 


j Familiar 


Some familiar I Some familiar 


Some familiar 


is the context? 








and some I and some 


and some 










unfamiliar j unfamiliar 


unfamiliar 


2. In how 


Single 


\ Single 


i More than one 


More than one j More than one 


More than one 


many different 


environment 


: environment 


j environment 


environment, j environment. 


environment. 


situations will 








indicating some j indicating some 


indicating some 


the task be 








transfer of skill I transfer of skill 


transfer of skill 


performed? 












KNOWLEDGE 












BASE: 












What will 












learners need 












to know to 












perform this 












task? 












I.What 


Simple 


: Simple 


j Growing 


Moderate store j Moderate store 


Good store of 


vocabulary 


vocabulary 


I vocabulary; 


I vocabulary. 


of vocabulary, j of vocabulary. 


vocabulary. 


related to the 




i with some 


i with a good 


including some j including some 


including some 


skill? related 




: multisyllablic 


\ store of 


unfamiliar and i unfamiliar and 


unfamiliar and 


to the subject 




j words 


j multisyllabic 


technical j technical 


technical 


area? 






j nontechnical 












: words 
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Analysis of Performance Task Requiring Effective Use of and EFF Standard 



1 Dimensions 


0-10 


11-20 


21-30 


31-40 


41-50 


51-60 


KNOWLEDGE 

BASE: 














2. What 


Minimal 


Familiarity with 


Familiarity with 


Famiiiarity with 


Familiarity with 


Familiarity with 


content 


familiarity 


a small store 


a good enough 


a good enough 


a good enough 


a range of facts. 


knowledge 


with content- 


of content- 


store of facts, 


store of facts, 


store of facts. 


operations. 


related to the 


reiated facts, 


related facts, 


operations. 


operations, 


operations. 


concepts. 


skill? Related 


operations, 


operations. 


concepts, rules, 


concepts, rules, 


concepts, rules. 


rules, protocols 


to the subject 


concepts, rules, 


concepts, rules. 


protocols and/ 


protocols and/ 


protocols and/ 


and/or 


area? 


protocols, and/ 


protocols, and/ 


or practices to 


or practices to 


or practices to 


practices. 




or practices 


or practices 


carry out 


carry out 


carry out the 


beyond the 


3. What 






the task 


the task 


task 


requirements 


strategies 
for organizing 




Only a small 


Some 


Some 


Range of 


of the task 


and applying 




number of 


strategies; 


strategies, 


strategies. 




content 




strategies; 


explanation; 


mostly simple. 


including a 




knowledge? 




including 


summary; 


a few "higher 


few “higher 








restatement. 


paraphrase; 


order”; pattern 


order”; pattern 








paraphrase, 


restatement; 


recognition; 


recognition; 




• Ability to 


Only a small 


and some 


use of 


ability to classi- 


identification 


Multiple 


recognize 


number of 


explanation. 


examples; 


fy/categorize 


of cause 


strategies. 


relationships 


strategies; 


Can use 


initial “pattern 


information; 


and effect 


some simple 


or connec- 


limited to 


examples 


recognition” 


some recogni- 


relationships; 


and some 


tions? 


simple recall of 






tion of cause 


ability to apply 


“higher order”; 




Information 






and effect 


new facts and 


pattern 










relationships; 


concepts to 


recognition; 


• Ability to 








explanation, 


prior experience 


identification 


create new 








interpretation, 


to create new 


of cause 


relationships 








translation, 


meaning; some 


and effect 


or connec- 








some 


generalization. 


relationships; 


tions? 








generalization, 


inference. 


use of prior 










inference. 


prediction 


knowledge and 










prediction 




application of 
rules/ 


• Ability to 


Little conscious 


Limited ability 


Some 


Ability to 


Ability to 


principles/etc. 


Identify 


ability to identify 


to identify 


conscious 


recognize 


identify 


to solve 


infomiation 


important 


important 


ability to identify 


and restate 


important 


problems; 


that is 


information 


information 


information that 


important 


information for 


some 


important to 






is relevant to a 


information for 


application 


generalization. 


the task/ 






clearly defined 


application to a 




inference. 


problem? 






purpose 


clearly defined 




prediction 










purpose 




ability to identify 








Some 


Good 


Good 


new information 


• Ability to 


Minimal 


Limited 


understanding 


understanding 


understanding 


and to 


understand 


understanding 


understanding 


of when 


of when 


of when 


self-monitor 


when infor- 


of when 


of when 


to apply 


to apply 


to apply 


comprehension. 


mation or 


to apply 


to apply 


information 


Information 


information 


Strong 


concepts 


information 


information 








apply? 












understanding 
of when identify 
important 
information for 
appiication 
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WORKSHEETS 

A. DEVELOPING A WELL-STRUCTURED PERFORMANCE TASK 

Name: Date: 

Description of the task: (Q6A and Q6B on the Reporting Form) 



What is the Standard addressed in the task? 



1. What are the components 
of performance for this Standard? 



How does the task incorporate each of the 
components? fQ6C on the Reporting Form) 
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2. Describe as fully as you can the task and its requirements in relation to 
the following Dimensions of Performance. (Q7, Reporting Form) Use the 
Task Template for guidance. 


Using the Task 
Template, assign 
a rating to the 
description. 


Complexity of the task: 




Context in which task will take place: 




Knowledge required 
for the task: 

1. vocabulary 
a. related to the 
Standard 


1a. 


1. 


b. related to the task/ 
content area 


lb. 




2. content knowledge 
a. related to the 
Standard 


2a. 


2. 


b. related to the task/ 
content area 


2b. 




3. strategies for organizing 
and applying content 
knowledge 
a. skill application 
strategies 


3a 


3. 


b. cognitive/ 
metacognitive 
strategies 


3b. 




3. Using the Task Template and the individual ratings given above, assign a 
rating to the overall task, within a five-point range. (Q8, Reporting Form) 
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4. Explain your reasons for rating the task at this point on the continuum, including the weighting 
(if any) of specific dimensions. (Q8, Reporting Form) 



5. Review the construction of the task. Indicate how the task 

• sufficiently focuses on the targeted Standard and its Components of Performance so that 
performance can be rated: 



• represents one instance of a meaningful, real-world use of the Standard: 



• has immediate use or high transfer value for learner(s): 



• is defined specifically enough so that the knowledge base requirements are clear: 



• identifies what evidence you will look at to see how well the Standard was used to carry out the task: 



ERIC 
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6. Look back at the Knowledge base requirements for this task. Did you include enough 
information (a rich enough description) to know what to look for: 



a. To document and assess student performance? 

b. To help us specify the template and build the continuum for this Standard? 



Knowledge Base Requirements 
for Task: 

1. Vocabulary 

a. related to the Standard 



What More Do You Need? 



b. related to the task/content area 



2. Content Knowledge 
a. related to the Standard 



b. related to the task/content area 



3. Strategies for Organizing 

and Applying Content Knowledge 
a. skill application strategies 



b. cognitive/metacognitive strategies 



7. As a teacher, how can you use this information to help you plan instruction? 
What will you pay attention to? 
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B. DESCRIBING STUDENT PERFORMANCE 

Now look at the descriptions of performance for each learner that you have collected (on the 
Observation Form and by other means). 

Use the table below to consider the following: 

8. Do the descriptions adequately address the knowledge base requirements of the task? 

(compare with Q2) 

9. Do descriptions adequately address (a) fluency, (b) independence, and (c) range of performance? 



10. Are the descriptions adequate to: 

a. Help you assess what the learner knows and can do? 

b. Help us build the performance continuum for this skill? (Q10 on the reporting form) 



Description of Performance 
on Standard for each Learner 


Adequate 


Inadequate 


What additional Information is needed 
in the description? 


Knowledge Base 

1. Vocabulary 

a. related to the Standard 

b. related to the task/ 
content area 
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Description of Performance 
on Standard for each Learner 



Adequate 



Inadequate 



What additional information is needed 
in the description? 



2. Content Knowledge 

a. related to the Standard 



b. related to the task/ 
content area 



3. Strategies for Organizing and 
Applying content knowledge 

a. skill application strategies 



b. cognitive/metacognitive 
strategies 



Fluency 



Independence 

Range 



11. How will you use this information to plan “Next Steps” for learners? 
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Student Documentation Form for EFF Standards 
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Equipped for the Future Standards 
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Appendix C 

Reporting 



Annotated Teacher Reporting Form 



Excerpts From Data Report on a Performance Task for Read With Understanding 
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EFF/NRS DATA COLLECTION PROJECT, 2000-2001 



TEACHER NAME:. 
PROGRAM NAME: 



Annotated Teacher Reporting Form 

EFF Field Development Reporting Form 2000-2001 



BEGIN DATE: 



PROGRAM/CLASS INFORMATION 

• %Q1- Which of the following terms does your 
program use to describe the location, 
administration and setting and of your program: 
(Please check all that apply with 
an "X” to the left of each item.) 

□ Rural (outside urban area, population < 2,500) 
n Urban (population >50,000) 

D Mixed rural/small cities 

□ Community college 

D Local Education Agency 

□ Community-based organization 

□ Correctional facility 

□ Workplace program 

□ Homeless program 

□ Family Literacy 

□ Other setting, please specify 



• %Q2. Which of the following terms describe the 
specific class or group of students you are reporting 
on: (Please check all that apply with an “X” to the left 
of each item.) 

□ Adult Basic Education (ABE) 

□ ESL 

□ Adult Secondary Education (ASE) — Family 
Literacy 

□ Workplace program 

□ Homeless program 

□ Correctional facilities 

□ Community corrections programs 

□ Other institutional programs 

• %Q3. Use the following NRS levels to describe the 
educational level of the students involved in the per- 



REPORT# 

formance task described in this report. 

□ Beginning ABE Literacy 

□ Beginning Basic Education 

□ Low Intermediate Basic Education 

□ High Intermediate Basic Education 

□ Low Adult Secondary Education 

□ High Adult Secondary Education 

□ Beginning ESL Literacy 

□ Beginning ESL 

□ Low Intermediate ESL 

□ High Intermediate ESL 

□ Low Advanced ESL 

□ High Advanced ESL 

□ 

Other: 

• %Q4- Which one individual standard are you doc- 
umenting in this report? (Please check with an “X” to 
the left of the item.) 

□ 1 . Read With Understanding 
D 2. Convey Ideas in Writing 

□ 3. Speak So Others Can Understand 

□ 4. Listen Actively 

D 5. Observe Critically 

□ 6. Use Math to Solve Problems and Communicate 

□ 7. Solve Problems and Make Decisions 

□ 8. Plan 

D 9. Cooperate with Others 

□ 10. Advocate and Influence 

□ 11. Resolve Conflict and Negotiate 

□ 12. Guide Others 

□ 13. Take Responsibility for Learning 

□ 14. Reflect and Evaluate 

□ 15. Learn Through Research 

□ 16. Use Information and Communications 

Technology 
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EFF Field Development Reporting Form 2000-2^)01 

• %Q5. Which Common Activity provides the con- 
text for the performance task described in this 
report? (Please check one with an “X” to the left of 
the item.) 

□ 1. Gather, Analyze and Use Information 

□ 2. Manage Resources 

□ 3. Work Within the Big Picture 

□ 4. Work Together 

□ 5. Provide Leadership 

□ 6. Guide and Support Others 

□ 7. Seek Guidance and Support From Others 
D 8. Develop and Express Sense of Self 

□ 9. Respect Others and Value Diversity 

□ 10. Exercise Rights and Responsibilities 

□ 11. Create and Pursue Vision and Goals 
D 1 2. Use Technology and Other Tools to 

Accomplish Goals 

□ 13. Keep Pace with Change 



IDENTIFYING AND RANKING THE TASK 

• %Q6, Provide a rich, detailed description of the 
task that requires use of the identified EFF standard, 
by answering the questions below: 

• %Q6A. What is the task? 

Enter the description of the performance task from 
the Worksheet. See point 5, p.7 in Chapter 1 for 
reminders. 

See Section 7, Examples i, 2, 3, to see how other teach’ 
ers answered this question, 

• %Q6B. What is the purpose of the task? 

Explain why and how it requires the learner to 
use the Standard. 

Enter the purpose from the Worksheet. 

See Section 7, Examples 1, 2, 3 

• %Q6C. Explain how this task requires learners to 
address each component of the standard. 

Again, refer to your Worksheet for this task. After 
you have determined that the task addresses each 
component, explain HOW. 



RATING THE TASK 

Use the task template to describe and provide a 
numerical rating for each dimension of the task. 

The task template is explained in Chapter 2: Creat- 
ing and Rating an EFF Performance Task. 

Refer to Section 7, Examples i,3, and 4 to see how 
other teachers responded to Q7A-C 

• %Q7A. Complexity of the task. 

DESCRIBE: 

Refer to Section 2 of the Worksheet^ where you recorded 
detailed descriptive information about the task. Enter 
the information about complexity here. 



RATING: 

Using the Task Template, rate the objective complexity 
of the task and enter the numerical value here. 

Do the same for Q7B-7C. 

• %Q7B. The context in which this task will take 
place. 

DESCRIBE: 



RATING: 

• %Q7C. What is the knowledge required for the 
task? Describe below, 7Cl -3, 



• Q7C1 . Vocabulary needed, related to the skill and 
to the subject area. 

DESCRIBE: 



RATING: 
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EFF Field Development Reporting Form 2000-2001 

• Q7C2. Content knowledge needed, related to the 
skill and to the subject area. 

DESCRIBE: 

RATING: 

• Q7C3. Strategies needed for organizing and 
applying the content knowledge. 

DESCRIBE: 

RATING: 

OVERALL TASK RATING: 

• %Q8. Provide a numerical rating of the overall 
task, within a five point range. Explain why you 
ranked the task here, including the weighting (if any) 
of specific dimensions. 

OVERALL RANK: 

REASONS: 

Refer to Example 1 for the way one teacher gave her 
rationale for the task rating. 

•Q9, Write about the learning activities that you 
and your students have been engaged in, to prepare 
for and perform the task described above in Ques- 
tions 6 -8. You may write in a journal format, with 
entries over time. What was your overall plan? What 
happened? How did it go? Were you pleased? Attach 
(to the hard copy of this report) the evidence 
of student performance related to this teaching/ 
learning interaction, including your written observa- 
tions of performance on observation worksheet. 

In Q9y write about what takes place^ both to prepare 
for the performance task and to carry out the perfor- 
mance task. 



Use the observation form to collect information on 
what learners know and how weU they perform, in 
relation to the knowledge required for the task. 
Describe what you see going on, writing notes 
directly on the observation form. You will use 
these forms to answer QIO for each student. 

Refer to Examples 4, 5, 6, 7, and 8, to see how other 
teachers wrote about their activities. 

Many of the Examples in Section 7 have artifacts 
attached. 



OBSERVATIONS OF STUDENT 
PERFORMANCE 

DATE OF COMPLETION OF 
PERFORMANCE TASK 

Q10. Answer the following questions about learner 
performance of the task described in Q6 and Q7. Use 
your Observation Worksheets and the Performance 
Template as your guide for describing performance 
and rating performance. Space is provided to report 
on three students. If you are reporting on more than 
three students, please refer to the technical instruc- 
tions for guidance in adding more students to this 
report. 

• %Q10 STUDENT 1: 

Make sure your artifacts and observation sheets 
for this student are labeled by this number. 

• Q10A. What does the learner know that allowed 
him/her to carry out the task as s/he did? Descrip- 
tion of vocabulary, content knowledge and strategies 
for organizing and applying content knowledge: 
Refer to Chapter 4, Knowledge Base, for guidance. 
See Examples 1,3, 4, 6, for iUustrations from last 
year’s data collection for all parts of QIO. 



RATING: 
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EFF Field Development Reporting Form 2000-2001 

Q10A1. How does this knowledge compare to the 
“knowledge required for the task” described in Q7C? 
Review what you wrote in Q7C about knowledge 
base requirements of the task, and your response to 
QIO A. Did the learner have and use all the knowl- 
edge necessary to accomplish the task? If not, what 
was missing? What does the learner still need to 
work on? 

• Q10B. How well did the learner use the skill 
described in the standard? 

• Q10B1. Description of fluency or ease of learner’s 
performance: 

RATING: 

• Q10B2. Description of independence of learner’s 
performance: 

RATING: 

• Q10C. Did the learner perform the task that was 
described in Q7A-C? 

Yes ' No 

IfnOy please answer the following questions: 

• Q10C1: In what ways was the task that the learner 
actually performed different from that described in 
Q7A-C? 

• Q10C2. Overall numerical rating of the task that 
the learner actually performed: 

• Q10D. Other comments on learner performance 
relative to the standard or to the task: 



REFLECTION AND EVALUATION 

The reflection and evaluation questions on this 
year’s form are not the same as last year’s. For 
examples of the ways that teachers wrote in this 
section, see Examples 3, 4, 9, and 10, Section 7. 

• Q11 A. Think back over the process of developing 
the performance task described in this report. 
Describe the extent to which the task template helps 
you in developing performance tasks. 

•Q11B. How did the process of rating the task 
go for you? Did your team agree with your initial 
rating? Was it difficult to come to consensus on a 
rating? Describe: 

• Q12. Think about the teaching and learning 
process that has been taking place. What is different 
from the way you usually teach? Is there a change in 
what is happening with your students? 

• Q13A. Think about the process of describing and 
rating student performance described in this report. 
Did the performance template help you observe and 
document what learners know more effectively? 
How? If not, why? 

• Q13B. Does the performance template help you 
compare one performance to another? 

• Q14. Think about the whole process of planning, 
teaching and documenting performance around a 
performance task. How did it work for you? Did you 
find it useful? Difficult?Challenging? What else? 
Please give us your honest and candid thoughts. 




74 



•Q15. Other comments: 



EFF/NRS DATA COLLECTION PROJECT. 2000-2001 



Excerpts from a data reporting form on a task developed for 
READ WITH UNDERSTANDING 

NRS Program Term: ESL 

NRS Level: high intermediate/low advanced 

Program Setting: community college 



• %Q6. Provide a rich, detailed description of the 
task that requires use of the identified EFF standard, 
by answering questions Q6A-D below: 

• %Q6A. Provide a clear, succinct statement of the 
task. 

Students will read newspaper classified ads adver- 
tising apartments for rent In their community. They 
will Identify the necessary information to answer 
questions they have previously formulated In class. 

• %Q6B. In questions Q6B1 and Q6B2 below, 
describe the meaningful use and transfer 
value of this task. 

• %Q6B1. Explain how the task represents a 
meaningful, real-world use of the standard. 

Students have expressed a concern about being 
able to find good, affordable housing. This task will 
help students better understand when reading rental 
housing classified ads, and thus be able to use the 
classified ads as a resource when searching for 
housing. 

• %Q6B2. Explain how students can apply in other 
situations (transfer) what they will learn by carrying 
out this task. 

Through this task students will gain a better 
understanding of the vocabulary, language, and 
abbreviations associated with classified ads. It is 
hoped that they will be able to use the classified ads 
in searching not only for rental housing, but other 
things as well (i.e. furniture, childcare, employment). 

• %Q6C. Explain how this task requires learners to 
address each component of the Standard. Determine 
the reading purpose: 

Students have already determined the purpose for 
reading In previous learning activities. In class dis- 



cussions students have addressed questions such 
as “What are the classified ads? How can they help 
me find housing?” Etc. They have also Identified 
what Information would be important to know when 
looking for rental housing, and have formulated 
questions to find out this Information. 

Select reading strategies appropriate to the purpose: 
When reading the classified ads for rental 
housing, students will need to select reading 
strategies such as scanning and reading for details 
to be able to find the Information to answer the 
questions (i.e., scanning — when looking at classified 
ads section titles, reading for details — looking for 
specific information such as amount of deposit and 
services Included In the rent). 

Monitor comprehension and adjust reading 
strategies: 

Students will search for the necessary Information to 
answer the questions and record their answers on a 
student-generated worksheet. Students will adjust 
their reading strategies until they are able to find the 
answers to these questions. Such adjustments may 
include using a dictionary, asking another student or 
the Instructor for help, determining meaning from 
context, etc. 

Analyze the information and reflect on its underlying 
meaning: 

Students will record the answers to the questions 
on a student-generated worksheet. By doing this 
they will determine what Important Information is 
given In the ad, and also what important Information 
is missing. 

Integrate it with prior knowledge to address reading 
purpose: 

The questions are evidence of the students’ prior 
knowledge regarding rental housing classified ads. 
Students are integrating prior knowledge with new 
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Excerpts from a data reporting form on a task developed for 
READ WITH UNDERSTANDING 



knowledge by finding the answers to the questions, 
and determining what important information is 
missing from the ads. 

• %Q6D. Explain what evidence you will look at to 
see how well the standard was used to carry out the 
task. 

When reading the classified ads, students will 
look for answers to questions they have previously 
formulated in class. They will then record their 
answers on a student-generated worksheet. Their 
answers will be evidence of how well the standard 
was used to carry out the task. 



RATING THE TASK 

• %Q7. Use the task template to describe and 
provide a numerical rating for each dimension of 
the task, below: 

• %Q7A. Complexity of the task 

Describe: The task is multi-stepped. It includes 
asking questions, reading, writing (recording 
answers), and analyzing. There is some teacher 
guidance and structure; however, the students 
themselves must formulate and answer the 
questions. 

RATING: 35 

• %Q7B. The context in which this task will take 
place 

Describe: The task takes place In two contexts: 

1) the classroom, and 2) the newspaper. The 
classroom is familiar, and the newspaper is 
somewhat familiar. 

RATING: 31 

• %Q7C. What is the knowledge required for the 
task? Describe below, 7C1-3. 

• %Q7C1. Vocabulary needed, related to the skill 
and to the subject area 

DESCRIBE: Skill: 1. Vocabulary related to asking 
and answering questions (i.e., how, what, how 
much, where, etc.) 2. Vocabulary related to reading 
strategies (i.e., pre-reading, scanning, detail, etc.) 



3. Vocabulary related to analyzing (i.e., information, 
relevant, important, missing, etc.) 

SUBJECT AREA: 1 . General vocabulary related 
to housing (i.e. rent, apartment, house, contract, 
amenities, etc.) 2.Vocabuiary specific to a 
newspaper (i.e., classified ads, sections, etc.) 

3. Vocabulary specific to rental housing classified 
ads. This also includes abbreviations (i.e. deposit, 
amenities, services included in rent, W/S/G, W/D, 
etc.) 4. Vocabulary specific to housing assistance 
programs in their community (i.e.. Housing Authority, 
Section 8, Public Housing, voucher, EHO, etc.) 

RATING: 37 

• %Q7C2. Content knowledge needed, related to 
the skill and to the subject area 

DESCRIBE: Skill: Ability to read in English; 
intermediate level of English comprehension; 
understanding of different reading strategies; ability 
to formulate and answer questions; understanding 
of the concept of abbreviations; ability to determine 
important/relevant information; and ability to think 
abstractly when determining what important/ 
relevant information is missing. 

SUBJECT AREA: General understanding of the 
rental housing process in the United States; 
understanding of a newspaper and where to locate 
the classified ads; understanding of how to use the 
classified ads (i.e. classification of different ads, 
responding to want ads, etc.); and understanding 
of the vocabulary, language, and abbreviations 
associated with rental housing classified ads. 

RATING: 39 

• %Q7C3. Strategies needed for organizing and 
applying the content knowledge 

DESCRIBE: Strategies needed include the ability to: 
Determine what is imporlant/relevant information, 
classify and categorize information, think abstractly 
(to determine what information is missing), 
understand the “intended” meaning in addition to 
the literal meaning (inferences), record and restate 
Information, link new knowledge to previous 
knowledge, and understand cultural protocols 
and ability to implement them. 

RATING: 39 
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Excerpts from a data reporting form on a task developed for 
READ WITH UNDERSTANDING 



OVERALL TASK RATING 

• %Q8. Provide a numerical rating of the overall 
task, within a five point range. Explain why you 
rated the task here, including the weighting (if any) 
of specific dimensions. 

OVERALL RATING: 35 - 40 

REASONS: I felt that this task should rate In the 
upper 30’s because It requires the students to think 
abstractly as they determine what Information is 
missing. Also, it requires the students to understand 
Inferences. 

TEACHING/IMPLEMENTING 

In Question 9, write about the learning activities that 
you and your students have been engaged in, to pre- 
pare for and perform the task described above. 

1 . JOURNAL — “Describe your home. How big is 
It? What color Is it? How did you find your home?” 
This opened up a discussion about the different 
resources students use for finding housing. This led 
to a discussion about the classified ads. Students 
expressed some of their frustrations when trying to 
read the classified ads. 

2. DISCUSSION — What are the classified ads? 
What do you already know about the classified ads? 
What do you not understand when using the classi- 
fied ads? Students discussed their experiences 
using the classified ads to buy things. They talked 
about the importance of being able to understand 
the language of the classified ads. 

3. JOURNAL — “Describe your ‘dream house.’ 
What would it look like? Where would it be locat- 
ed?” This opened up a discussion about vocabulary 
related to housing. From there we discussed 
vocabulary and abbreviations used in classified ads 
for rental housing. Students then determined what 
information would be important to know when 
looking for rental housing. 

4. HOMEWORK ASSIGNMENT — Students wrote 
questions to find out the information they previously 
determined to be important information when look- 
ing for rental housing. They were informed that we 
would use these questions in an exercise practicing 
reading classified ads for rental housing. 

5. DISCUSSION — The class discussed their expe- 
rience with the homework assignment of writing 



questions. They talked about what other information 
might be important to know when looking for rental 
housing. This led to a discussion about American 
culture and the protocols and expectations when 
renting in the United States. 

DATE OF COMPLETION OF PERFORMANCE TASK: 
March 12, 2001 

OBSERVATIONS OF STUDENT 
PERFORMANCE 

Answer the following questions about learner perfor- 
mance of the task described in Q6 and Q7. Use the 
Observation Worksheets and Performance Template 
as your guide for describing performance and rating 
performance. 

• %Q10 STUDENT 1: JC [Full report includes 
performance data for two more students] 

• %Q10A. What does the learner know that 
allowed him/her to carry out the task as s/he did? 
Description of vocabulary, content knowledge and 
strategies for organizing and applying content 
knowledge: 

JC understands vocabulary related to asking 
questions (i.e., how, what, where, when, etc.) She 
demonstrated this understanding in the homework 
assignment of writing questions to inquire about 
an apartment for rent as well as in our class 
discussions. 

She understands the concept of reading 
strategies and has much experience with this as 
she has a college degree from her native country. 
Although she understands and can do “scanning” 
and “reading for detail,” she lacks some of the 
English vocabulary to express the idea. This was 
evident in some of the class discussions when she 
asked for clarification of the meanings of some of 
these words (i.e. scanning). She, however, easily 
used different reading strategies in her work. 

Once again she understands the concepts of 
analyzing, and was able to analyze, but lacks some 
of the English vocabulary. She asked for clarification 
of the meanings of these words (i.e., relevant). 

She understands general vocabulary related 
to housing as she is currently renting an apartment 
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(i.e., rent, deposit, house, apartment, contract). 

She understands some vocabulary related 
to the newspaper as she reads the newspaper daily 
in class. She is able to identify different sections of 
the newspaper and navigates well within the 
newspaper. She was able to identify the classified 
ads section without any difficulty. 

She has enough vocabulary specific to rental 
housing to be able to ask and answer questions 
(i.e., deposit, rent, contract, lease). However, some 
of the regional vocabulary was new to her (i.e., 
porch vs. patio vs. deck, utility room, trailer vs. 
mobile home vs. manufactured home). The 
abbreviations were new to her and she asked 
several questions regarding abbreviations. 

She was unfamiliar with the vocabulary 
specific to housing assistance programs in the 
community. The concept of housing assistance 
was also new to her as she commented that such 
programs are not available in her native country. 
During class discussions she had several questions 
regarding housing assistance in the U.S. 

RATING: 40 

• %Q10A1. How does this knowledge compare to 
the “knowledge required for the task” described in 
Q7C? 

JC’s knowledge was sufficient and even above what 
was required for the task. This was demonstrated in 
her ability, ease, and quality of work in completing 
the task. She lacked some of the technical vocabu- 
lary needed, but was able to learn it. 

• %Q10B. How well did the learner use the skill 
described in the standard? 

JC used the skills described in the standard very 
well. She was able to determine reading purpose as 
she participated in class discussions. She was able 
to adjust her reading strategies as needed to help 
her understanding. I observed her use her dictio- 
nary, ask others for clarification, and relate new 
information to what she already knew (i.e., “Oh, that 
is like...”). She was able to analyze information and 
determine relevant information as well as determine 
what information was missing. This was evident in 
her answers on the worksheet. She was able to 
make inferences when determining missing informa- 
tion and also able to determine cause and effect 
(i.e., ‘This apartment includes electricity and water, 
therefore, the rent must be higher”). Also, that she 
was able to formulate questions inquiring about an 



apartment for rent, was evidence of prior 
knowledge. She was also able to understand 
humor during class discussions about rental 
housing. For example, when discussing the 
homework assignment to ask questions, one of 
the questions was related to the number of kids 
allowed. One student said, “Do you allow kids as 
well as horses?” She indicated that she understood 
the humor — that kids are also baby goats. 

• %Q10B1. Description of fluency or ease of learn- 
er’s performance: 

This task did not seem to require much effort from 
JC. She appeared to easily adjust reading strategies 
as needed and was able to analyze information. In 
my observations she moved easily from one part of 
the task to another. She was one of the first 
students to complete the task. 

RATING: 55 

• %Q10B2. Description of independence of learn- 
er s performance: 

JC was able to complete the task without 
assistance from the instructor. When she was 
unsure of vocabulary she used her dictionary or 
asked other students. She demonstrated no 
difficulty in being able to start and end the task. 

She was one of the first students to complete 
the task. 

RATING: 61 

• %Q10C. Did the learner perform the task that 
was described in Q7A-C? 

X Yes No 

Please describe the evidence that you have collected: 

1. Homework assignment to ask questions to 
inquire about an apartment for rent. 

2. Completed worksheet. 

3. Additional questions which were part of the 
assignment, but not part of the task. Answers to 
these questions demonstrate an understanding 
of vocabulary. 
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